调度中心日志删除优化,改为分页获取ID并根据ID删除的方式,避免批量删除海量日志导致死锁问题

master
xuxueli 5 years ago
parent ada47e5aef
commit e73be8175c
  1. 5
      doc/XXL-JOB官方文档.md
  2. 9
      xxl-job-admin/src/main/java/com/xxl/job/admin/controller/JobLogController.java
  3. 10
      xxl-job-admin/src/main/java/com/xxl/job/admin/dao/XxlJobLogDao.java
  4. 40
      xxl-job-admin/src/main/resources/mybatis-mapper/XxlJobLogMapper.xml

@ -1590,8 +1590,8 @@ Tips: 历史版本(V1.3.x)目前已经Release至稳定版本, 进入维护阶段
- 19、调度中心移除SQL中的 "now()" 函数;集群部署时不再依赖DB时钟,仅需要保证调度中心应用节点时钟一致即可;
- 20、xxl-rpc服务端线程优化,降低线程内存开销;
- 21、调度中心回调API服务改为restful方式;
- 22、[ING]调度日志优化:支持设置日志保留天数,过期日志天维度记录报表,并清理;调度报表汇总实时数据和报表
- 23、[ING]调度中心日志删除,改为分页获取ID,根据ID删除的方式
- 22、调度中心日志删除优化,改为分页获取ID并根据ID删除的方式,避免批量删除海量日志导致死锁问题
- 23、[ING]调度日志优化:支持设置日志保留天数,过期日志天维度记录报表,并清理;调度报表汇总实时数据和报表
@ -1632,6 +1632,7 @@ Tips: 历史版本(V1.3.x)目前已经Release至稳定版本, 进入维护阶段
- 34、通讯调整;双向HTTP,回调和其他API自定义AccessToken,Restful,执行器复用容器端口;
- 35、父子任务参数传递;流程任务等,透传动态参数;
- 36、任务操作API服务调整为和回调服务一致,降低接入成本;
- 37、新增执行器描述、任务描述属性;
## 七、其他

@ -216,7 +216,14 @@ public class JobLogController {
return new ReturnT<String>(ReturnT.FAIL_CODE, I18nUtil.getString("joblog_clean_type_unvalid"));
}
xxlJobLogDao.clearLog(jobGroup, jobId, clearBeforeTime, clearBeforeNum);
List<Long> logIds = null;
do {
logIds = xxlJobLogDao.findClearLogIds(jobGroup, jobId, clearBeforeTime, clearBeforeNum, 1000);
if (logIds!=null && logIds.size()>0) {
xxlJobLogDao.clearLog(logIds);
}
} while (logIds!=null && logIds.size()>0);
return ReturnT.SUCCESS;
}

@ -46,10 +46,12 @@ public interface XxlJobLogDao {
public List<Map<String, Object>> triggerCountByDay(@Param("from") Date from,
@Param("to") Date to);
public int clearLog(@Param("jobGroup") int jobGroup,
@Param("jobId") int jobId,
@Param("clearBeforeTime") Date clearBeforeTime,
@Param("clearBeforeNum") int clearBeforeNum);
public List<Long> findClearLogIds(@Param("jobGroup") int jobGroup,
@Param("jobId") int jobId,
@Param("clearBeforeTime") Date clearBeforeTime,
@Param("clearBeforeNum") int clearBeforeNum,
@Param("pagesize") int pagesize);
public int clearLog(@Param("logIds") List<Long> logIds);
public List<Long> findFailJobLogIds(@Param("pagesize") int pagesize);

@ -186,8 +186,8 @@
ORDER BY triggerDay
</select>
<delete id="clearLog" >
delete from xxl_job_log
<select id="findClearLogIds" resultType="long" >
SELECT id FROM xxl_job_log
<trim prefix="WHERE" prefixOverrides="AND | OR" >
<if test="jobGroup gt 0">
AND job_group = #{jobGroup}
@ -200,22 +200,32 @@
</if>
<if test="clearBeforeNum gt 0">
AND id NOT in(
SELECT id FROM(
SELECT id FROM xxl_job_log AS t
<trim prefix="WHERE" prefixOverrides="AND | OR" >
<if test="jobGroup gt 0">
AND t.job_group = #{jobGroup}
</if>
<if test="jobId gt 0">
AND t.job_id = #{jobId}
</if>
</trim>
ORDER BY t.trigger_time desc
LIMIT 0, #{clearBeforeNum}
) t1
SELECT id FROM(
SELECT id FROM xxl_job_log AS t
<trim prefix="WHERE" prefixOverrides="AND | OR" >
<if test="jobGroup gt 0">
AND t.job_group = #{jobGroup}
</if>
<if test="jobId gt 0">
AND t.job_id = #{jobId}
</if>
</trim>
ORDER BY t.trigger_time desc
LIMIT 0, #{clearBeforeNum}
) t1
)
</if>
</trim>
order by id asc
LIMIT #{pagesize}
</select>
<delete id="clearLog" >
delete from xxl_job_log
WHERE id in
<foreach collection="logIds" item="item" open="(" close=")" separator="," >
#{item}
</foreach>
</delete>
<select id="findFailJobLogIds" resultType="long" >

Loading…
Cancel
Save