• 确认报表统计方案
  • 统计维度
  • 数据收集
  • 数据展示
  • 复杂SQL的编写,Echarts报表的使用

报表统计方案

  • 统计维度:
    • 统计数值:总阅读数,总点赞数,今日阅读数,今日点赞数,今日预计阅读数,今日预计阅读增长
    • 统计报表:30天阅读/点赞趋势图、文档阅读量排名(热门文章)、文档点赞量排名(优质文章)
  • 业务表统计:所有报表数据都是从业务表直接获取的
    • 优点:实时性好(数据准确)、工作量
    • 缺点:对业务表性能有影响,有些统计无法实现(比如上面的今日阅读数,点赞数就没法统计)
  • 中间表统计:定时将业务表数据汇总到中间表,报表数据从中间表获取
    • 优点:性能好,可实现多功能统计
    • 缺点:工作量大,步骤多容易出错

电子书快照表设计

  • 概念:快照
  • 电子书快照表:一天存储一次快照

因为我们这个需要每天的统计数据,所以我们的快照是一天一次,也就是说每天要存储一次(条)当时的电子书的信息,我们下面就是电子书快照表的设计sql:

  1. drop table if exists `ebook_snapshot`;
  2. create table `ebook_snapshot` (
  3. `id` bigint auto_increment not null comment 'id',
  4. `ebook_id` bigint not null default 0 comment '电子书id',
  5. `date` date not null comment '快照日期',
  6. `view_count` int not null default 0 comment '阅读数',
  7. `vote_count` int not null default 0 comment '点赞数',
  8. `view_increase` int not null default 0 comment '阅读增长',
  9. `vote_increase` int not null default 0 comment '点赞增长',
  10. primary key (`id`),
  11. unique key `ebook_id_date_unique` (`ebook_id`,`date`)
  12. ) engine=innodb default charset=utf8mb4 comment='电子书快照表'

这样的设计保证了每一天每一门电子书应该有一条数据,同时使用ebook_id和date作为唯一键,然后去gennerator-config.xml当中去最后添加下面的代码,并执行mybatis-gennerator生成持久层代码:

  1. <table tableName="ebook_snapshot"/>

电子书快照收集脚本编写

  • 从业务表收集数据的SQL尽量简单,不要影响业务表性能
  • 快照分成两个部分
    • 总量:总阅读数,总点赞数(总量简单,只需要从业务表直接把当前最新的数据把它放进来就可以了)
    • 增量:今日阅读数,今日点赞数(稍微复杂一些,把今天的总量减去昨天的总量,就是今天的增量)

我们的收集策略也不能简单的就是在23:59:59这个时间点去收集,而是应该每个小时收集一次,然后每天的第一次收集向表当中插入数据,后续的都来更新这条数据的值即可。所以我们应该先写insert,然后再update,需要更新的字段其中view_count和vote_count只需要从业务表当中取最新的数据即可,而view_increase和vote_increase需要一点计算,需要获取到昨天的view_count、vote_count和今天的view_count、vote_count,然后做差值。

下面我们来书写一下跑批的sql的四个步骤:

  • 第一步,为所有电子书生成一条今天的记录,如果还没有
  • 第二步,把今天的阅读数、点赞数全部更新进来
  • 第三步,根据今天和昨天的总数据来计算出这个阅读的增长

对应的sql如下:

  1. # 插入记录(如果今天没有数据的话)
  2. insert into ebook_snapshot(ebook_id, `date`, view_count, vote_count, view_increase, vote_increase)
  3. select t1.id, curdate(), 0, 0, 0, 0
  4. from ebook t1
  5. where not exists(select 1
  6. from ebook_snapshot t2
  7. where t1.id = t2.ebook_id
  8. and t2.`date` = curdate());
  9. # 更新view_count和vote_count
  10. update ebook_snapshot t1, ebook t2
  11. set t1.view_count = t2.view_count,t1.vote_count = t2.vote_count
  12. where t1.`date`=curdate()
  13. and t1.ebook_id = t2.id;
  14. #获取昨天的数据,两者做差
  15. update ebook_snapshot t1 left join (select ebook_id, view_count, vote_count from ebook_snapshot
  16. where `date` = date_sub(curdate(), interval 1 day)) t2
  17. on t1.ebook_id=t2.ebook_id
  18. set t1.view_increase = (t1.view_count-ifnull(t2.view_count, 0)),
  19. t1.vote_increase = (t1.vote_count-ifnull(t2.vote_count, 0))
  20. where t1.`date` = curdate();

完成电子书快照任务

首先如果我们要在mybatis当中执行多个sql的话,我们需要在数据库的配置当中添加allowMultiQueries=true这样的参数,如下:

  1. # 增加数据库连接
  2. spring.datasource.url=jdbc:mysql://rm-hp3f05153598ck3e5vo.mysql.huhehaote.rds.aliyuncs.com/wikidev?characterEncoding=UTF8&autoReconnect=true&serverTimezone=Asia/Shanghai&allowMultiQueries=true

然后我们就开始倒着展示代码:

  1. <?xml version="1.0" encoding="UTF-8" ?>
  2. <!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
  3. <mapper namespace="com.taopoppy.wiki.mapper.EbookSnapshotMapperCust" >
  4. <!--
  5. # 方案一(ID不连续):
  6. # 删除今天的数据
  7. # 为所有的电子书生成一条今天的记录
  8. # 更新总阅读数、总点赞数
  9. # 更新今日阅读数、今日点赞数
  10. # 方案二(ID连续):
  11. # 为所有的电子书生成一条今天的记录,如果还没有
  12. # 更新总阅读数、总点赞数
  13. # 更新今日阅读数、今日点赞数
  14. -->
  15. <update id="genSnapshot">
  16. insert into ebook_snapshot(ebook_id, `date`, view_count, vote_count, view_increase, vote_increase)
  17. select t1.id, curdate(), 0, 0, 0, 0
  18. from ebook t1
  19. where not exists(select 1
  20. from ebook_snapshot t2
  21. where t1.id = t2.ebook_id
  22. and t2.`date` = curdate());
  23. update ebook_snapshot t1, ebook t2
  24. set t1.view_count = t2.view_count,
  25. t1.vote_count = t2.vote_count
  26. where t1.`date` = curdate()
  27. and t1.ebook_id = t2.id;
  28. update ebook_snapshot t1 left join (select ebook_id, view_count, vote_count
  29. from ebook_snapshot
  30. where `date` = date_sub(curdate(), interval 1 day)) t2
  31. on t1.ebook_id = t2.ebook_id
  32. set t1.view_increase = (t1.view_count - ifnull(t2.view_count, 0)),
  33. t1.vote_increase = (t1.vote_count - ifnull(t2.vote_count, 0))
  34. where t1.`date` = curdate();
  35. </update>
  36. </mapper>
  1. package com.taopoppy.wiki.mapper;
  2. public interface EbookSnapshotMapperCust {
  3. public void genSnapshot();
  4. }
  1. package com.taopoppy.wiki.service;
  2. import com.taopoppy.wiki.mapper.EbookSnapshotMapperCust;
  3. import org.springframework.stereotype.Service;
  4. import javax.annotation.Resource;
  5. @Service
  6. public class EbookSnapshotService {
  7. @Resource
  8. private EbookSnapshotMapperCust ebookSnapshotMapperCust;
  9. public void genSnapshot() {
  10. ebookSnapshotMapperCust.genSnapshot();
  11. }
  12. }

最后我们在定时器当中去执行快照的程序:

  1. package com.taopoppy.wiki.job;
  2. import com.taopoppy.wiki.service.EbookSnapshotService;
  3. import com.taopoppy.wiki.util.SnowFlake;
  4. import org.slf4j.Logger;
  5. import org.slf4j.LoggerFactory;
  6. import org.slf4j.MDC;
  7. import org.springframework.scheduling.annotation.Scheduled;
  8. import org.springframework.stereotype.Component;
  9. import javax.annotation.Resource;
  10. @Component
  11. public class EbookSnapshotJob {
  12. private static final Logger LOG = LoggerFactory.getLogger(EbookSnapshotJob.class);
  13. @Resource
  14. private EbookSnapshotService ebookSnapshotService;
  15. @Resource
  16. private SnowFlake snowFlake;
  17. /**
  18. * 自定义cron表达式跑批
  19. * 只有等上一次执行完成,下一次才会在下一个时间点执行,错过就错过
  20. */
  21. @Scheduled(cron = "0 0/1 * * * ?")
  22. public void doSnapshot() {
  23. // 增加日志流水号
  24. MDC.put("LOG_ID", String.valueOf(snowFlake.nextId()));
  25. LOG.info("生成今日电子书快照开始");
  26. Long start = System.currentTimeMillis();
  27. ebookSnapshotService.genSnapshot();
  28. LOG.info("生成今日电子书快照结束,耗时:{}毫秒", System.currentTimeMillis() - start);
  29. }
  30. }

首页统计功能

1. 后端代码

首页包含数值统计和30天趋势图,所以我们从controller开始看看后端代码:

  1. @RestController
  2. @RequestMapping("/ebook-snapshot")
  3. public class EbookSnapshotController {
  4. @Resource
  5. private EbookSnapshotService ebookSnapshotService;
  6. @GetMapping("/get-statistic")
  7. public CommonResp getStatistic() {
  8. List<StatisticResp> statisticResp = ebookSnapshotService.getStatistic();
  9. CommonResp<List<StatisticResp>> commonResp = new CommonResp<>();
  10. commonResp.setContent(statisticResp);
  11. return commonResp;
  12. }
  13. @GetMapping("/get-30-statistic")
  14. public CommonResp get30Statistic() {
  15. List<StatisticResp> statisticResp = ebookSnapshotService.get30Statistic();
  16. CommonResp<List<StatisticResp>> commonResp = new CommonResp<>();
  17. commonResp.setContent(statisticResp);
  18. return commonResp;
  19. }
  20. }
  1. @Service
  2. public class EbookSnapshotService {
  3. @Resource
  4. private EbookSnapshotMapperCust ebookSnapshotMapperCust;
  5. public void genSnapshot() {
  6. ebookSnapshotMapperCust.genSnapshot();
  7. }
  8. /**
  9. * 获取首页数值数据:总阅读数、总点赞数、今日阅读数、今日点赞数、今日预计阅读数、今日预计阅读增长
  10. */
  11. public List<StatisticResp> getStatistic() {
  12. return ebookSnapshotMapperCust.getStatistic();
  13. }
  14. /**
  15. * 30天数值统计
  16. */
  17. public List<StatisticResp> get30Statistic() {
  18. return ebookSnapshotMapperCust.get30Statistic();
  19. }
  20. }
  1. public interface EbookSnapshotMapperCust {
  2. public void genSnapshot();
  3. List<StatisticResp> getStatistic();
  4. List<StatisticResp> get30Statistic();
  5. }
  1. package com.taopoppy.wiki.resp;
  2. import com.fasterxml.jackson.annotation.JsonFormat;
  3. import java.util.Date;
  4. public class StatisticResp {
  5. @JsonFormat(pattern="MM-dd", timezone = "GMT+8")
  6. private Date date;
  7. private int viewCount;
  8. private int voteCount;
  9. private int viewIncrease;
  10. private int voteIncrease;
  11. public Date getDate() {
  12. return date;
  13. }
  14. public void setDate(Date date) {
  15. this.date = date;
  16. }
  17. public int getViewCount() {
  18. return viewCount;
  19. }
  20. public void setViewCount(int viewCount) {
  21. this.viewCount = viewCount;
  22. }
  23. public int getVoteCount() {
  24. return voteCount;
  25. }
  26. public void setVoteCount(int voteCount) {
  27. this.voteCount = voteCount;
  28. }
  29. public int getViewIncrease() {
  30. return viewIncrease;
  31. }
  32. public void setViewIncrease(int viewIncrease) {
  33. this.viewIncrease = viewIncrease;
  34. }
  35. public int getVoteIncrease() {
  36. return voteIncrease;
  37. }
  38. public void setVoteIncrease(int voteIncrease) {
  39. this.voteIncrease = voteIncrease;
  40. }
  41. @Override
  42. public String toString() {
  43. return "StatisticResp{" +
  44. "date=" + date +
  45. ", viewCount=" + viewCount +
  46. ", voteCount=" + voteCount +
  47. ", viewIncrease=" + viewIncrease +
  48. ", voteIncrease=" + voteIncrease +
  49. '}';
  50. }
  51. }
  1. <?xml version="1.0" encoding="UTF-8" ?>
  2. <!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
  3. <mapper namespace="com.taopoppy.wiki.mapper.EbookSnapshotMapperCust" >
  4. <!-- 获取首页数值数据:总阅读数、总点赞数、今日阅读数、今日点赞数、今日预计阅读数、今日预计阅读增长 -->
  5. <select id="getStatistic" resultType="com.taopoppy.wiki.resp.StatisticResp">
  6. select
  7. t1.`date` as `date`,
  8. sum(t1.view_count) as viewCount,
  9. sum(t1.vote_count) as voteCount,
  10. sum(t1.view_increase) as viewIncrease,
  11. sum(t1.vote_increase) as voteIncrease
  12. from
  13. ebook_snapshot t1
  14. where
  15. t1.`date` >= date_sub(curdate(), interval 1 day)
  16. group by
  17. t1.`date`
  18. order by
  19. t1.`date` asc;
  20. </select>
  21. <select id="get30Statistic" resultType="com.taopoppy.wiki.resp.StatisticResp">
  22. select
  23. t1.`date` as `date`,
  24. sum(t1.view_increase) as viewIncrease,
  25. sum(t1.vote_increase) as voteIncrease
  26. from
  27. ebook_snapshot t1
  28. where
  29. t1.`date` between date_sub(curdate(), interval 30 day) and date_sub(curdate(), interval 1 day)
  30. group by
  31. t1.`date`
  32. order by
  33. t1.`date` asc;
  34. </select>
  35. </mapper>

2. 前端代码

前端是需要去使用echarts的,所以我们可以将echarts.min.js直接拷到项目当中去,然后在index.html去引入即可,具体的统计页面我们写在了the-welcome.vue当中,代码如下:

  1. <template>
  2. <div>
  3. <div class="tip">
  4. <div><b>示例网站说明(网站实现的功能课程都有手把手教):</b></div>
  5. <div>1. 统计数据是真实的,一分钟左右延时,<b>用到了定时器、复杂SQL统计、echarts</b></div>
  6. <div>2. 有文档被别人点赞,你也会收到实时通知哦!<b>用到了websocket、异步化、RocketMQ、防重设计</b></div>
  7. <div>3. 文档树可无限级扩展,支持文字、图片、视频。<b>用到了无限级树设计知识</b></div>
  8. <div>4. 登录后可看到更多菜单。<b>用到了单点登录、前后端登录拦截、安全性设计</b></div>
  9. <div>课程技术栈思维导图,
  10. <a href="http://www.jiawablog.com/detail?id=152412053087326208" target="_blank">
  11. 点击查看:SpringBoot+Vue3
  12. </a>
  13. </div>
  14. <div>你也想有个WIKI知识库吗?,<b>配套视频课程</b>
  15. <a href="https://coding.imooc.com/class/474.html" target="_blank">
  16. 《SpringBoot知识体系+Vue3全家桶 前后端分离 实战WIKI知识库系统》
  17. </a>
  18. </div>
  19. </div>
  20. <a-row>
  21. <a-col :span="24">
  22. <a-card>
  23. <a-row>
  24. <a-col :span="8">
  25. <a-statistic title="总阅读量" :value="statistic.viewCount">
  26. <template #suffix>
  27. <UserOutlined />
  28. </template>
  29. </a-statistic>
  30. </a-col>
  31. <a-col :span="8">
  32. <a-statistic title="总点赞量" :value="statistic.voteCount">
  33. <template #suffix>
  34. <like-outlined />
  35. </template>
  36. </a-statistic>
  37. </a-col>
  38. <a-col :span="8">
  39. <a-statistic title="点赞率" :value="statistic.voteCount / statistic.viewCount * 100"
  40. :precision="2"
  41. suffix="%"
  42. :value-style="{ color: '#cf1322' }">
  43. <template #suffix>
  44. <like-outlined />
  45. </template>
  46. </a-statistic>
  47. </a-col>
  48. </a-row>
  49. </a-card>
  50. </a-col>
  51. </a-row>
  52. <br>
  53. <a-row :gutter="16">
  54. <a-col :span="12">
  55. <a-card>
  56. <a-row>
  57. <a-col :span="12">
  58. <a-statistic title="今日阅读" :value="statistic.todayViewCount" style="margin-right: 50px">
  59. <template #suffix>
  60. <UserOutlined />
  61. </template>
  62. </a-statistic>
  63. </a-col>
  64. <a-col :span="12">
  65. <a-statistic title="今日点赞" :value="statistic.todayVoteCount">
  66. <template #suffix>
  67. <like-outlined />
  68. </template>
  69. </a-statistic>
  70. </a-col>
  71. </a-row>
  72. </a-card>
  73. </a-col>
  74. <a-col :span="12">
  75. <a-card>
  76. <a-row>
  77. <a-col :span="12">
  78. <a-statistic
  79. title="预计今日阅读"
  80. :value="statistic.todayViewIncrease"
  81. :value-style="{ color: '#0000ff' }"
  82. >
  83. <template #suffix>
  84. <UserOutlined />
  85. </template>
  86. </a-statistic>
  87. </a-col>
  88. <a-col :span="12">
  89. <a-statistic
  90. title="预计今日阅读增长"
  91. :value="statistic.todayViewIncreaseRateAbs"
  92. :precision="2"
  93. suffix="%"
  94. class="demo-class"
  95. :value-style="statistic.todayViewIncreaseRate < 0 ? { color: '#3f8600' } : { color: '#cf1322' }"
  96. >
  97. <template #prefix>
  98. <arrow-down-outlined v-if="statistic.todayViewIncreaseRate < 0"/>
  99. <arrow-up-outlined v-if="statistic.todayViewIncreaseRate >= 0"/>
  100. </template>
  101. </a-statistic>
  102. </a-col>
  103. </a-row>
  104. </a-card>
  105. </a-col>
  106. </a-row>
  107. <br>
  108. <a-row>
  109. <a-col :span="24" id="main-col">
  110. <div id="main" style="width: 100%;height:300px;"></div>
  111. </a-col>
  112. </a-row>
  113. </div>
  114. </template>
  115. <script lang="ts">
  116. import { defineComponent, ref, onMounted } from 'vue'
  117. import axios from 'axios';
  118. declare let echarts: any;
  119. export default defineComponent({
  120. name: 'the-welcome',
  121. setup () {
  122. const statistic = ref();
  123. statistic.value = {};
  124. const getStatistic = () => {
  125. axios.get('/ebook-snapshot/get-statistic').then((response) => {
  126. const data = response.data;
  127. if (data.success) {
  128. const statisticResp = data.content;
  129. statistic.value.viewCount = statisticResp[1].viewCount;
  130. statistic.value.voteCount = statisticResp[1].voteCount;
  131. statistic.value.todayViewCount = statisticResp[1].viewIncrease;
  132. statistic.value.todayVoteCount = statisticResp[1].voteIncrease;
  133. // 按分钟计算当前时间点,占一天的百分比
  134. const now = new Date();
  135. const nowRate = (now.getHours() * 60 + now.getMinutes()) / (60 * 24);
  136. // console.log(nowRate)
  137. statistic.value.todayViewIncrease = parseInt(String(statisticResp[1].viewIncrease / nowRate));
  138. // todayViewIncreaseRate:今日预计增长率
  139. statistic.value.todayViewIncreaseRate = (statistic.value.todayViewIncrease - statisticResp[0].viewIncrease) / statisticResp[0].viewIncrease * 100;
  140. statistic.value.todayViewIncreaseRateAbs = Math.abs(statistic.value.todayViewIncreaseRate);
  141. }
  142. });
  143. };
  144. const init30DayEcharts = (list: any) => {
  145. // 发布生产后出现问题:切到别的页面,再切回首页,报表显示不出来
  146. // 解决方法:把原来的id=main的区域清空,重新初始化
  147. const mainDom = document.getElementById('main-col');
  148. if (mainDom) {
  149. mainDom.innerHTML = '<div id="main" style="width: 100%;height:300px;"></div>';
  150. }
  151. // 基于准备好的dom,初始化echarts实例
  152. const myChart = echarts.init(document.getElementById('main'));
  153. const xAxis = [];
  154. const seriesView = [];
  155. const seriesVote = [];
  156. for (let i = 0; i < list.length; i++) {
  157. const record = list[i];
  158. xAxis.push(record.date);
  159. seriesView.push(record.viewIncrease);
  160. seriesVote.push(record.voteIncrease);
  161. }
  162. // 指定图表的配置项和数据
  163. const option = {
  164. title: {
  165. text: '30天趋势图'
  166. },
  167. tooltip: {
  168. trigger: 'axis'
  169. },
  170. legend: {
  171. data: ['总阅读量', '总点赞量']
  172. },
  173. grid: {
  174. left: '1%',
  175. right: '3%',
  176. bottom: '3%',
  177. containLabel: true
  178. },
  179. toolbox: {
  180. feature: {
  181. saveAsImage: {}
  182. }
  183. },
  184. xAxis: {
  185. type: 'category',
  186. boundaryGap: false,
  187. data: xAxis
  188. },
  189. yAxis: {
  190. type: 'value'
  191. },
  192. series: [
  193. {
  194. name: '总阅读量',
  195. type: 'line',
  196. // stack: '总量', 不堆叠
  197. data: seriesView,
  198. smooth: true
  199. },
  200. {
  201. name: '总点赞量',
  202. type: 'line',
  203. // stack: '总量', 不堆叠
  204. data: seriesVote,
  205. smooth: true
  206. }
  207. ]
  208. };
  209. // 使用刚指定的配置项和数据显示图表。
  210. myChart.setOption(option);
  211. };
  212. const get30DayStatistic = () => {
  213. axios.get('/ebook-snapshot/get-30-statistic').then((response) => {
  214. const data = response.data;
  215. if (data.success) {
  216. const statisticList = data.content;
  217. init30DayEcharts(statisticList)
  218. }
  219. });
  220. };
  221. const testEcharts = () => {
  222. // 基于准备好的dom,初始化echarts实例
  223. const myChart = echarts.init(document.getElementById('main'));
  224. // 指定图表的配置项和数据
  225. const option = {
  226. title: {
  227. text: 'ECharts 入门示例'
  228. },
  229. tooltip: {},
  230. legend: {
  231. data:['销量']
  232. },
  233. xAxis: {
  234. data: ["衬衫","羊毛衫","雪纺衫","裤子","高跟鞋","袜子"]
  235. },
  236. yAxis: {},
  237. series: [{
  238. name: '销量',
  239. type: 'bar',
  240. data: [5, 20, 36, 10, 10, 20]
  241. }]
  242. };
  243. // 使用刚指定的配置项和数据显示图表。
  244. myChart.setOption(option);
  245. };
  246. onMounted(() => {
  247. getStatistic();
  248. // testEcharts();
  249. get30DayStatistic();
  250. });
  251. return {
  252. statistic
  253. }
  254. }
  255. });
  256. </script>
  257. <style scoped>
  258. .tip {
  259. padding: 10px 5px;
  260. margin-bottom: 20px;
  261. border: 1px solid transparent;
  262. background: linear-gradient(white,white) padding-box,repeating-linear-gradient(-45deg, black 0, black 25%, white 0, white 50%) 0/.6em .6em;
  263. animation:ants 12s linear infinite;
  264. }
  265. .tip b{
  266. color: red;
  267. }
  268. </style>