需求分析

产品经理召集开会，文章审核功能已经交付了，文章也能正常发布审核。对于上次提出的自管理敏感词也很满意，这次会议核心的内容如下：

  • 文章中包含的图片要识别文字，过滤掉图片文字的敏感词

8.2)图片文字识别

什么是OCR?

OCR （Optical Character Recognition，光学字符识别）是指电子设备（例如扫描仪或数码相机）检查纸上打印的字符，通过检测暗、亮的模式确定其形状，然后用字符识别方法将形状翻译成计算机文字的过程

方案 说明
百度OCR 收费
Tesseract-OCR Google维护的开源OCR引擎，支持Java，Python等语言调用
Tess4J 封装了Tesseract-OCR ，支持Java调用

8.3)Tess4j案例

①：创建项目导入tess4j对应的依赖

  1. <dependency>
  2.     <groupId>net.sourceforge.tess4j</groupId>
  3.     <artifactId>tess4j</artifactId>
  4.     <version>4.1.1</version>
  5. </dependency>

②：导入中文字体库， 把资料中的tessdata文件夹拷贝到自己的工作空间下

图片识别文字审核敏感词 - 图1

③：编写测试类进行测试

  1. package com.heima.tess4j;
  3. import net.sourceforge.tess4j.ITesseract;
  4. import net.sourceforge.tess4j.Tesseract;
  6. import java.io.File;
  8. public class Application {
  10.     public static void main(String[] args) {
  11.         try {
  12.             //获取本地图片
  13.             File file = new File("D:\\26.png");
  14.             //创建Tesseract对象
  15.             ITesseract tesseract = new Tesseract();
  16.             //设置字体库路径
  17.             tesseract.setDatapath("D:\\workspace\\tessdata");
  18.             //中文识别
  19.             tesseract.setLanguage("chi_sim");
  20.             //执行ocr识别
  21.             String result = tesseract.doOCR(file);
  22.             //替换回车和tal键  使结果为一行
  23.             result = result.replaceAll("\\r|\\n","-").replaceAll(" ","");
  24.             System.out.println("识别的结果为："+result);
  25.         } catch (Exception e) {
  26.             e.printStackTrace();
  27.         }
  28.     }
  29. }

8.4)管理敏感词和图片文字识别集成到文章审核

①：在heima-leadnews-common中创建工具类，简单封装一下tess4j

需要先导入pom

  1. <dependency>
  2.     <groupId>net.sourceforge.tess4j</groupId>
  3.     <artifactId>tess4j</artifactId>
  4.     <version>4.1.1</version>
  5. </dependency>

工具类

  1. package com.heima.common.tess4j;
  3. import lombok.Getter;
  4. import lombok.Setter;
  5. import net.sourceforge.tess4j.ITesseract;
  6. import net.sourceforge.tess4j.Tesseract;
  7. import net.sourceforge.tess4j.TesseractException;
  8. import org.springframework.boot.context.properties.ConfigurationProperties;
  9. import org.springframework.stereotype.Component;
  11. import java.awt.image.BufferedImage;
  13. @Getter
  14. @Setter
  15. @Component
  16. @ConfigurationProperties(prefix = "tess4j")
  17. public class Tess4jClient {
  19.     private String dataPath;
  20.     private String language;
  22.     public String doOCR(BufferedImage image) throws TesseractException {
  23.         //创建Tesseract对象
  24.         ITesseract tesseract = new Tesseract();
  25.         //设置字体库路径
  26.         tesseract.setDatapath(dataPath);
  27.         //中文识别
  28.         tesseract.setLanguage(language);
  29.         //执行ocr识别
  30.         String result = tesseract.doOCR(image);
  31.         //替换回车和tal键  使结果为一行
  32.         result = result.replaceAll("\\r|\\n", "-").replaceAll(" ", "");
  33.         return result;
  34.     }
  36. }

在spring.factories配置中添加该类,完整如下：

  1. org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  2.   com.heima.common.exception.ExceptionCatch,\
  3.   com.heima.common.swagger.SwaggerConfiguration,\
  4.   com.heima.common.swagger.Swagger2Configuration,\
  5.   com.heima.common.aliyun.GreenTextScan,\
  6.   com.heima.common.aliyun.GreenImageScan,\
  7.   com.heima.common.tess4j.Tess4jClient

②：在heima-leadnews-wemedia中的配置中添加两个属性

  1. tess4j:
  2.   data-path: D:\workspace\tessdata
  3.   language: chi_sim

③：在WmNewsAutoScanServiceImpl中的handleImageScan方法上添加如下代码

  1. try {
  2.     for (String image : images) {
  3.         byte[] bytes = fileStorageService.downLoadFile(image);
  5.         //图片识别文字审核---begin-----
  7.         //从byte[]转换为butteredImage
  8.         ByteArrayInputStream in = new ByteArrayInputStream(bytes);
  9.         BufferedImage imageFile = ImageIO.read(in);
  10.         //识别图片的文字
  11.         String result = tess4jClient.doOCR(imageFile);
  13.         //审核是否包含自管理的敏感词
  14.         boolean isSensitive = handleSensitiveScan(result, wmNews);
  15.         if(!isSensitive){
  16.             return isSensitive;
  17.         }
  19.         //图片识别文字审核---end-----
  22.         imageList.add(bytes);
  24.     } 
  25. }catch (Exception e){
  26.     e.printStackTrace();
  27. }

最后附上文章审核的完整代码如下：

  1. package com.heima.wemedia.service.impl;
  3. import com.alibaba.fastjson.JSONArray;
  4. import com.baomidou.mybatisplus.core.toolkit.Wrappers;
  5. import com.heima.apis.article.IArticleClient;
  6. import com.heima.common.aliyun.GreenImageScan;
  7. import com.heima.common.aliyun.GreenTextScan;
  8. import com.heima.common.tess4j.Tess4jClient;
  9. import com.heima.file.service.FileStorageService;
  10. import com.heima.model.article.dtos.ArticleDto;
  11. import com.heima.model.common.dtos.ResponseResult;
  12. import com.heima.model.wemedia.pojos.WmChannel;
  13. import com.heima.model.wemedia.pojos.WmNews;
  14. import com.heima.model.wemedia.pojos.WmSensitive;
  15. import com.heima.model.wemedia.pojos.WmUser;
  16. import com.heima.utils.common.SensitiveWordUtil;
  17. import com.heima.wemedia.mapper.WmChannelMapper;
  18. import com.heima.wemedia.mapper.WmNewsMapper;
  19. import com.heima.wemedia.mapper.WmSensitiveMapper;
  20. import com.heima.wemedia.mapper.WmUserMapper;
  21. import com.heima.wemedia.service.WmNewsAutoScanService;
  22. import lombok.extern.slf4j.Slf4j;
  23. import org.apache.commons.lang3.StringUtils;
  24. import org.springframework.beans.BeanUtils;
  25. import org.springframework.beans.factory.annotation.Autowired;
  26. import org.springframework.scheduling.annotation.Async;
  27. import org.springframework.stereotype.Service;
  28. import org.springframework.transaction.annotation.Transactional;
  30. import javax.imageio.ImageIO;
  31. import java.awt.image.BufferedImage;
  32. import java.io.ByteArrayInputStream;
  33. import java.util.*;
  34. import java.util.stream.Collectors;
  37. @Service
  38. @Slf4j
  39. @Transactional
  40. public class WmNewsAutoScanServiceImpl implements WmNewsAutoScanService {
  42.     @Autowired
  43.     private WmNewsMapper wmNewsMapper;
  45.     /**
  46.      * 自媒体文章审核
  47.      *
  48.      * @param id 自媒体文章id
  49.      */
  50.     @Override
  51.     @Async  //标明当前方法是一个异步方法
  52.     public void autoScanWmNews(Integer id) {
  54. //        int a = 1/0;
  56.         //1.查询自媒体文章
  57.         WmNews wmNews = wmNewsMapper.selectById(id);
  58.         if (wmNews == null) {
  59.             throw new RuntimeException("WmNewsAutoScanServiceImpl-文章不存在");
  60.         }
  62.         if (wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())) {
  63.             //从内容中提取纯文本内容和图片
  64.             Map<String, Object> textAndImages = handleTextAndImages(wmNews);
  66.             //自管理的敏感词过滤
  67.             boolean isSensitive = handleSensitiveScan((String) textAndImages.get("content"), wmNews);
  68.             if(!isSensitive) return;
  70.             //2.审核文本内容  阿里云接口
  71.             boolean isTextScan = handleTextScan((String) textAndImages.get("content"), wmNews);
  72.             if (!isTextScan) return;
  74.             //3.审核图片  阿里云接口
  75.             boolean isImageScan = handleImageScan((List<String>) textAndImages.get("images"), wmNews);
  76.             if (!isImageScan) return;
  78.             //4.审核成功，保存app端的相关的文章数据
  79.             ResponseResult responseResult = saveAppArticle(wmNews);
  80.             if (!responseResult.getCode().equals(200)) {
  81.                 throw new RuntimeException("WmNewsAutoScanServiceImpl-文章审核，保存app端相关文章数据失败");
  82.             }
  83.             //回填article_id
  84.             wmNews.setArticleId((Long) responseResult.getData());
  85.             updateWmNews(wmNews, (short) 9, "审核成功");
  87.         }
  88.     }
  90.     @Autowired
  91.     private WmSensitiveMapper wmSensitiveMapper;
  93.     /**
  94.      * 自管理的敏感词审核
  95.      * @param content
  96.      * @param wmNews
  97.      * @return
  98.      */
  99.     private boolean handleSensitiveScan(String content, WmNews wmNews) {
  101.         boolean flag = true;
  103.         //获取所有的敏感词
  104.         List<WmSensitive> wmSensitives = wmSensitiveMapper.selectList(Wrappers.<WmSensitive>lambdaQuery().select(WmSensitive::getSensitives));
  105.         List<String> sensitiveList = wmSensitives.stream().map(WmSensitive::getSensitives).collect(Collectors.toList());
  107.         //初始化敏感词库
  108.         SensitiveWordUtil.initMap(sensitiveList);
  110.         //查看文章中是否包含敏感词
  111.         Map<String, Integer> map = SensitiveWordUtil.matchWords(content);
  112.         if(map.size() >0){
  113.             updateWmNews(wmNews,(short) 2,"当前文章中存在违规内容"+map);
  114.             flag = false;
  115.         }
  117.         return flag;
  118.     }
  120.     @Autowired
  121.     private IArticleClient articleClient;
  123.     @Autowired
  124.     private WmChannelMapper wmChannelMapper;
  126.     @Autowired
  127.     private WmUserMapper wmUserMapper;
  129.     /**
  130.      * 保存app端相关的文章数据
  131.      *
  132.      * @param wmNews
  133.      */
  134.     private ResponseResult saveAppArticle(WmNews wmNews) {
  136.         ArticleDto dto = new ArticleDto();
  137.         //属性的拷贝
  138.         BeanUtils.copyProperties(wmNews, dto);
  139.         //文章的布局
  140.         dto.setLayout(wmNews.getType());
  141.         //频道
  142.         WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId());
  143.         if (wmChannel != null) {
  144.             dto.setChannelName(wmChannel.getName());
  145.         }
  147.         //作者
  148.         dto.setAuthorId(wmNews.getUserId().longValue());
  149.         WmUser wmUser = wmUserMapper.selectById(wmNews.getUserId());
  150.         if (wmUser != null) {
  151.             dto.setAuthorName(wmUser.getName());
  152.         }
  154.         //设置文章id
  155.         if (wmNews.getArticleId() != null) {
  156.             dto.setId(wmNews.getArticleId());
  157.         }
  158.         dto.setCreatedTime(new Date());
  160.         ResponseResult responseResult = articleClient.saveArticle(dto);
  161.         return responseResult;
  163.     }
  166.     @Autowired
  167.     private FileStorageService fileStorageService;
  169.     @Autowired
  170.     private GreenImageScan greenImageScan;
  172.     @Autowired
  173.     private Tess4jClient tess4jClient;
  175.     /**
  176.      * 审核图片
  177.      *
  178.      * @param images
  179.      * @param wmNews
  180.      * @return
  181.      */
  182.     private boolean handleImageScan(List<String> images, WmNews wmNews) {
  184.         boolean flag = true;
  186.         if (images == null || images.size() == 0) {
  187.             return flag;
  188.         }
  190.         //下载图片 minIO
  191.         //图片去重
  192.         images = images.stream().distinct().collect(Collectors.toList());
  194.         List<byte[]> imageList = new ArrayList<>();
  196.         try {
  197.             for (String image : images) {
  198.                 byte[] bytes = fileStorageService.downLoadFile(image);
  200.                 //图片识别文字审核---begin-----
  202.                 //从byte[]转换为butteredImage
  203.                 ByteArrayInputStream in = new ByteArrayInputStream(bytes);
  204.                 BufferedImage imageFile = ImageIO.read(in);
  205.                 //识别图片的文字
  206.                 String result = tess4jClient.doOCR(imageFile);
  208.                 //审核是否包含自管理的敏感词
  209.                 boolean isSensitive = handleSensitiveScan(result, wmNews);
  210.                 if(!isSensitive){
  211.                     return isSensitive;
  212.                 }
  214.                 //图片识别文字审核---end-----
  217.                 imageList.add(bytes);
  219.             }
  220.         }catch (Exception e){
  221.             e.printStackTrace();
  222.         }
  225.         //审核图片
  226.         try {
  227.             Map map = greenImageScan.imageScan(imageList);
  228.             if (map != null) {
  229.                 //审核失败
  230.                 if (map.get("suggestion").equals("block")) {
  231.                     flag = false;
  232.                     updateWmNews(wmNews, (short) 2, "当前文章中存在违规内容");
  233.                 }
  235.                 //不确定信息  需要人工审核
  236.                 if (map.get("suggestion").equals("review")) {
  237.                     flag = false;
  238.                     updateWmNews(wmNews, (short) 3, "当前文章中存在不确定内容");
  239.                 }
  240.             }
  242.         } catch (Exception e) {
  243.             flag = false;
  244.             e.printStackTrace();
  245.         }
  246.         return flag;
  247.     }
  249.     @Autowired
  250.     private GreenTextScan greenTextScan;
  252.     /**
  253.      * 审核纯文本内容
  254.      *
  255.      * @param content
  256.      * @param wmNews
  257.      * @return
  258.      */
  259.     private boolean handleTextScan(String content, WmNews wmNews) {
  261.         boolean flag = true;
  263.         if ((wmNews.getTitle() + "-" + content).length() == 0) {
  264.             return flag;
  265.         }
  266.         try {
  267.             Map map = greenTextScan.greeTextScan((wmNews.getTitle() + "-" + content));
  268.             if (map != null) {
  269.                 //审核失败
  270.                 if (map.get("suggestion").equals("block")) {
  271.                     flag = false;
  272.                     updateWmNews(wmNews, (short) 2, "当前文章中存在违规内容");
  273.                 }
  275.                 //不确定信息  需要人工审核
  276.                 if (map.get("suggestion").equals("review")) {
  277.                     flag = false;
  278.                     updateWmNews(wmNews, (short) 3, "当前文章中存在不确定内容");
  279.                 }
  280.             }
  281.         } catch (Exception e) {
  282.             flag = false;
  283.             e.printStackTrace();
  284.         }
  286.         return flag;
  288.     }
  290.     /**
  291.      * 修改文章内容
  292.      *
  293.      * @param wmNews
  294.      * @param status
  295.      * @param reason
  296.      */
  297.     private void updateWmNews(WmNews wmNews, short status, String reason) {
  298.         wmNews.setStatus(status);
  299.         wmNews.setReason(reason);
  300.         wmNewsMapper.updateById(wmNews);
  301.     }
  303.     /**
  304.      * 1。从自媒体文章的内容中提取文本和图片
  305.      * 2.提取文章的封面图片
  306.      *
  307.      * @param wmNews
  308.      * @return
  309.      */
  310.     private Map<String, Object> handleTextAndImages(WmNews wmNews) {
  312.         //存储纯文本内容
  313.         StringBuilder stringBuilder = new StringBuilder();
  315.         List<String> images = new ArrayList<>();
  317.         //1。从自媒体文章的内容中提取文本和图片
  318.         if (StringUtils.isNotBlank(wmNews.getContent())) {
  319.             List<Map> maps = JSONArray.parseArray(wmNews.getContent(), Map.class);
  320.             for (Map map : maps) {
  321.                 if (map.get("type").equals("text")) {
  322.                     stringBuilder.append(map.get("value"));
  323.                 }
  325.                 if (map.get("type").equals("image")) {
  326.                     images.add((String) map.get("value"));
  327.                 }
  328.             }
  329.         }
  330.         //2.提取文章的封面图片
  331.         if (StringUtils.isNotBlank(wmNews.getImages())) {
  332.             String[] split = wmNews.getImages().split(",");
  333.             images.addAll(Arrays.asList(split));
  334.         }
  336.         Map<String, Object> resultMap = new HashMap<>();
  337.         resultMap.put("content", stringBuilder.toString());
  338.         resultMap.put("images", images);
  339.         return resultMap;
  341.     }
  342. }

9)文章详情-静态文件生成

9.1)思路分析

文章端创建app相关文章时，生成文章详情静态页上传到MinIO中

图片识别文字审核敏感词 - 图2

9.2)实现步骤

1.新建ArticleFreemarkerService创建静态文件并上传到minIO中

  1. package com.heima.article.service;
  3. import com.heima.model.article.pojos.ApArticle;
  5. public interface ArticleFreemarkerService {
  7.     /**
  8.      * 生成静态文件上传到minIO中
  9.      * @param apArticle
  10.      * @param content
  11.      */
  12.     public void buildArticleToMinIO(ApArticle apArticle,String content);
  13. }

实现

  1. package com.heima.article.service.impl;
  3. import com.alibaba.fastjson.JSON;
  4. import com.alibaba.fastjson.JSONArray;
  5. import com.baomidou.mybatisplus.core.toolkit.Wrappers;
  6. import com.heima.article.mapper.ApArticleContentMapper;
  7. import com.heima.article.service.ApArticleService;
  8. import com.heima.article.service.ArticleFreemarkerService;
  9. import com.heima.file.service.FileStorageService;
  10. import com.heima.model.article.pojos.ApArticle;
  11. import freemarker.template.Configuration;
  12. import freemarker.template.Template;
  13. import lombok.extern.slf4j.Slf4j;
  14. import org.apache.commons.lang3.StringUtils;
  15. import org.springframework.beans.BeanUtils;
  16. import org.springframework.beans.factory.annotation.Autowired;
  17. import org.springframework.scheduling.annotation.Async;
  18. import org.springframework.stereotype.Service;
  19. import org.springframework.transaction.annotation.Transactional;
  21. import java.io.ByteArrayInputStream;
  22. import java.io.InputStream;
  23. import java.io.StringWriter;
  24. import java.util.HashMap;
  25. import java.util.Map;
  27. @Service
  28. @Slf4j
  29. @Transactional
  30. public class ArticleFreemarkerServiceImpl implements ArticleFreemarkerService {
  32.     @Autowired
  33.     private ApArticleContentMapper apArticleContentMapper;
  35.     @Autowired
  36.     private Configuration configuration;
  38.     @Autowired
  39.     private FileStorageService fileStorageService;
  41.     @Autowired
  42.     private ApArticleService apArticleService;
  44.     /**
  45.      * 生成静态文件上传到minIO中
  46.      * @param apArticle
  47.      * @param content
  48.      */
  49.     @Async
  50.     @Override
  51.     public void buildArticleToMinIO(ApArticle apArticle, String content) {
  52.         //已知文章的id
  53.         //4.1 获取文章内容
  54.         if(StringUtils.isNotBlank(content)){
  55.             //4.2 文章内容通过freemarker生成html文件
  56.             Template template = null;
  57.             StringWriter out = new StringWriter();
  58.             try {
  59.                 template = configuration.getTemplate("article.ftl");
  60.                 //数据模型
  61.                 Map<String,Object> contentDataModel = new HashMap<>();
  62.                 contentDataModel.put("content", JSONArray.parseArray(content));
  63.                 //合成
  64.                 template.process(contentDataModel,out);
  65.             } catch (Exception e) {
  66.                 e.printStackTrace();
  67.             }
  69.             //4.3 把html文件上传到minio中
  70.             InputStream in = new ByteArrayInputStream(out.toString().getBytes());
  71.             String path = fileStorageService.uploadHtmlFile("", apArticle.getId() + ".html", in);
  74.             //4.4 修改ap_article表，保存static_url字段
  75.             apArticleService.update(Wrappers.<ApArticle>lambdaUpdate().eq(ApArticle::getId,apArticle.getId())
  76.                     .set(ApArticle::getStaticUrl,path));
  79.         }
  80.     }
  82. }

2.在ApArticleService的saveArticle实现方法中添加调用生成文件的方法

  1. /**
  2.      * 保存app端相关文章
  3.      * @param dto
  4.      * @return
  5.      */
  6. @Override
  7. public ResponseResult saveArticle(ArticleDto dto) {
  9.     //        try {
  10.     //            Thread.sleep(3000);
  11.     //        } catch (InterruptedException e) {
  12.     //            e.printStackTrace();
  13.     //        }
  14.     //1.检查参数
  15.     if(dto == null){
  16.         return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
  17.     }
  19.     ApArticle apArticle = new ApArticle();
  20.     BeanUtils.copyProperties(dto,apArticle);
  22.     //2.判断是否存在id
  23.     if(dto.getId() == null){
  24.         //2.1 不存在id  保存  文章  文章配置  文章内容
  26.         //保存文章
  27.         save(apArticle);
  29.         //保存配置
  30.         ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId());
  31.         apArticleConfigMapper.insert(apArticleConfig);
  33.         //保存 文章内容
  34.         ApArticleContent apArticleContent = new ApArticleContent();
  35.         apArticleContent.setArticleId(apArticle.getId());
  36.         apArticleContent.setContent(dto.getContent());
  37.         apArticleContentMapper.insert(apArticleContent);
  39.     }else {
  40.         //2.2 存在id   修改  文章  文章内容
  42.         //修改  文章
  43.         updateById(apArticle);
  45.         //修改文章内容
  46.         ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers.<ApArticleContent>lambdaQuery().eq(ApArticleContent::getArticleId, dto.getId()));
  47.         apArticleContent.setContent(dto.getContent());
  48.         apArticleContentMapper.updateById(apArticleContent);
  49.     }
  51.     //异步调用 生成静态文件上传到minio中
  52.     articleFreemarkerService.buildArticleToMinIO(apArticle,dto.getContent());
  55.     //3.结果返回  文章的id
  56.     return ResponseResult.okResult(apArticle.getId());
  57. }

3.文章微服务开启异步调用

图片识别文字审核敏感词 - 图3