Pipeline logging:
Jenkins core:
The current default approach of storing everything into the filesystem is the main reason why Jenkins does not fit the "Cloud Native" model, with features like HA, zero downtime, or pay per use. While there are plenty of plugins that implement parts of this vision, this becomes cumbersome to configure and a usability nightmare for users, as JEP-300 has pointed out. Going towards a model where cloud services are consumed where it makes sense, the overall complexity on operating Jenkins in a cloud or containerized environment is greatly reduced. Other related projects include Jenkins X and Jenkins Evergreen which would greatly benefit from a Cloud Native storage for Jenkins.
There are several clear areas open for improvement, which are summarized here and will be detailed in future documents. A mayor pain point is the usage of local disk as all-purpose storage, which causes issues running on containerized or distributed environments, requiring highly performant filesystems, and all the configuration pain like initial sizing and resizing with downtime. We believe that by using cloud provided services the overall usability, performance and scalability can be improved while enabling new demanded functionality.
You can find more information about Pluggable Storage and priorities in this blogpost.
Below you can find a summary of ongoing activities and their current status:
| Type / Status | Comment | Implementation(s) |
|---|---|---|
Artifacts |
Fully delivered, with support for uploading artifacts directly from agents. Related JEPs: JEP-202. |
Artifact Manager on S3, Azure Artifact Manager, <More implementations> |
Credentials |
Completed before the JEP process was introduced. |
Kubernetes Credentials Provider, AWS Secrets Manager Credentials Provider, <More implementations> |
Build logs |
Pipeline Log Storage API and reference implementations are available for preview, only Jenkins Pipeline job types are supported. Related JEP: JEP-210. Jenkins core APIs and reference implementations have not been merged/released yet, but prototypes are available for evaluation. Related JEPs: JEP-207, JEP-212 |
|
System logs |
Jenkins supports custom log appenders using standard |
Syslog logger, non-Jenkins implementation |
Task logs |
Storage of system logs and various tasks (e.g. agent connection or SCM polling) |
N/A |
Configurations |
Largely replaced by the Configuration as Code plugin which allows storing Jenkins configurations in SCM or other locations without a database. Related JEPs: JEP-213 |
N/A |
Test Results |
API is available in JUnit Plugin. |
|
Code Coverage results |
Planned only for plugins based on Code Coverage API which unifies the storage implementation. See Runs for other coverage report types. |
N/A |
Builds/Runs |
Storage of full build records in an external database. Includes storing build data which is not otherwise listed (such as logs or test results). |
N/A |
Jobs |
Storage of Job configurations and job-specific metadata in an external database. Existing plugins like Jenkins Pipeline and JobDSL partially address this case by keeping configurations in SCM. |
N/A |
Fingerprints |
Jenkins 2.252+ include the external fingerprint storage API which can be consumed by plugins. More info: GSoC Project Page Related JEPs: JEP-226 |
|
Workspaces |
Proposed as a GSoC 2019 project: Cloud Features for External Workspace Manager Plugin |
N/A |
The list above is not complete. Other storage types may be considered according to the feedback. You can find more information about Pluggable Storage and priorities in this blogpost.
There are many existing plugins allowing to upload and download artifacts from external storage (e.g. S3, Artifactory, Publish over SFTP, etc., etc.), but there are no plugins which can do it transparently without using new steps. In many cases the artifacts also get uploaded through the Jenkins controller, and it increases load on the system. It would be great if there was a layer which would allow storing artifacts externally when using common steps like Archive Artifacts.
Jenkins 2.118+ offers an extended jenkins.util.VirtualFile API which allows implementing external artifact managers using the ArtifactManagerFactory extension point.
Implementation example(s):
Related JEPs:
Logs disk usage causes the same issues previously mentioned for artifacts. Plus a external focused log storage such as AWS CloudWatch Logs allows demanded features as centralized log management, log retention policies, advanced querying, etc. There are already options to externally ship the logs to a backend, or plugins that would do that like the aws-cloudwatch-logs-publisher-plugin, but there is no integrated way to both send and display logs from external log storage. The External log storage work is tracked as issue:38313[].
Reference implementation(s):
Related JEPs:
Although configurations are not big, externalizing them is a critical task
for getting highly-available or disposable Jenkins controllers.
There are many ways to store configurations in Jenkins,
but 95% of cases are covered by the XmlFile layer which
serializes objects to disk and reads them using the XStream library.
Externalizing these XmlFiles would be a great step forward.
There are several prototypes for externalizing configurations, e.g. in DotCI. There are also other implementations which could be upstreamed to the Jenkins core.
Related JEPs:
In Credentials Plugin 1.15+ there is a CredentialsProvider extension point which allows referencing and resolving external credentials. This engine allows implementing external credentials for plugins implementing Credentials API..
Implementation example(s):
Other credentials API in Jenkins (like jenkinsdoc:hudson.util.Secret) are not supported.
In common CI/CD use-cases a lot of the space is being consumed by test reports.
This data is stored within JENKINS_HOME,
and the current storage format requires huge overheads when retrieving statistics and, especially, trends.
In order to display trends, each report has to be loaded and then processed in-memory.
The main purpose of externalising Test Results is to optimize Jenkins logic by querying the desired data from specialized external storages, e.g. from Document-based databases like Elasticsearch. According to the current plan, JUnit Plugin will be extended in order to support such external storage in its APIs being widely used by test reporting plugins.
Status:
A SQL implementation is available Junit SQL Storage plugin.
Please try it out, report issues to GitHub and general feedback to GitHub#142.
The fingerprints are stored within JENKINS_HOME inside a local XML-based database.
Externalizing fingerprints decreases the dependence of Jenkins on the physical disk storage of the controller, and allows configuring of cloud storages which can be cheaper, and more reliable.
Another advantage is that it would allow tracing fingerprints across Jenkins instances and the entire CI/CD flow.
Status:
In progress
Related JEP: JEP-226: External Fingerprint Storage
Reference Implementation: Redis Fingerprint Storage Plugin