This is a brief, but hopefully useful documentation of Archiva. I've just put this up to organize my thoughts while I am doing MRM-409 and it might be of help to other developers too.

Repository Scanning and Indexing

Assumption:
-default-archiva.xml is used for the configuration

Classes

Below are some of the important classes of Repository Scanning:

Repository Content Consumers (KnownRepositoryContentConsumer)

This is configured in archiva.xml, under <repositoryScanning>.

The Process

1. User clicks 'Scan Repository Now' in the Repositories page.
2. Repository scanning is triggered.
3. Start scanning:
   • DefaultRepositoryScanner gathers the consumers (KnownContentConsumers and InvalidContentConsumers) from the config file. RepositoryScannerInstance is added as a DirectoryWalkListener to the plexus-utils DirectoryWalker. Start of scan is fired.
   • Every file discovered will be checked if it is in the includes or excludes patterns that is set. If it doesn't exist in both, then it would be excluded. If it is included, then it will be processed by the consumers. Each consumer performs a different action in its processFile(...) method.
4. Saving the artifact to the database is performed in the ArtifactUpdateDatabaseConsumer. An ArchivaArtifact, which has an ArchivaArtifactModel attribute, is constructed. The attributes of the ArchivaArtifactModel are gathered from the artifact itself e.g. groupId, artifactId, version came from the artifact's filepath.
5. Indexing the artifact happens in the IndexContentConsumer, wherein an index record which contains the details of the artifact plus its contents. Please note that in the default-archiva.xml, the bundled files are not included in the indexable-content fileType pattern.¹ 
6. Once the repository scanning is finished, the scan statistics (number of files discovered, the consumers used, duration of the scan, the repository scanned, etc.) is listed or displayed in the console.
7. User performs a search:
   • User types the query string and hits the Search button.
   • Archiva then searches its indices for the query string and returns the search results.
   • The user can click on an artifact to browse it. Actually, what the user browses is the pom. At the back-end, Archiva checks if the project model is already in the database. If it is not, then archiva constructs the ArchivaProjectModel object and saves it to the database.¹ Once it is already in the database, the pom info or artifact is displayed.

¹ This causes the problem of different values when the actual pom file is read. The pom file may be invalid (e.g. it might have different versions as in the case of commons-dbcp-1.0 in MRM-376) and wasn't detected when it was added to the database (MRM-409).

Finding an Artifact

1. The user browses for an artifact he/she wants to locate in the repositories.
2. Archiva calculates the checksum for the artifact to be searched.
3. The database is searched for the matching checksum using the ArtifactsByChecksumConstraint (search all artifacts where the calculated checksum matches either a SHA1 or MD5 checksum of an artifact in the database)
   
   
   

Registry Listeners

A RegistryListener (plexus-registry) is an interface that receives notification for every change in the Registry. There are a handlful of classes in Archiva that implements this and performs some processes every time there's a change in the configuration.

Database Scanning

The database is scanned and specific consumers process these artifacts.

Database Consumers

There are 2 types of database consumers:

1. Unprocessed consumers - consumers for those artifacts already in the index that haven't been processed yet, meaning the details about the artifact are not yet processed and stored in the database
2. Cleanup consumers - consumers for cleaning up the database

These consumers are configured in archiva.xml, under <databaseScanning>. Below are the different types of Database Consumers:

Repository Purge

Remove old snapshots from the managed repository based on a criteria: By Number of Days Old and By Retention Count. There is also the option to enable or disable the cleanup of released snapshots from the repository.

Classes

Below are the classes for Repository Purge:

Configuration (for Archiva Users)

1. To enable repository purge, add "repository-purge" in the <knownContentConsumers> section of the archiva.xml. The RepositoryPurgeConsumer will be executed when repository scanning is started.
2. The user can choose whether to purge the repository of snapshots older by a specific number of days OR to purge the repository of snapshots but retaining a specific number of that snapshot. This can be configured by specifying specific values in the "Repository Purge By Days Older Than" or "Repository Purge By Retention Count" fields in the Add/Edit Repository page. By default, these has "100" and "2" values respectively. If "Repository Purge By Days Older" is NOT EQUAL TO 0 (zero), then that would be the criteria used for the repository purge. Otherwise, if it is EQUAL TO 0 (zero) then the "Repository Purge By Retention Count" criteria is used instead.
3. To enable/disable the cleanup of released snapshots in the repository, the user can opt to check or uncheck the "Delete Released Snapshots" option in the Add/Edit Repository page.
   

The Process

1. RepositoryPurgeConsumer is executed during repository scanning. Only those "artifact" file types are consumed (<fileType> with "artifact" id in archiva.xml).
2. The consumer will check the if the deleteReleasedSnapshots field (in RepositoryConfiguration) is enabled. If so, then it will execute CleanupReleasedSnapshotsRepositoryPurge.
   • CleanupReleasedSnapshotsRepositoryPurge will remove all released snapshots from the repository. For example: 1.2, 1.3-SNAPSHOT and 1.3 exists for artifactX in the repo. 1.3-SNAPSHOT will be removed since 1.3 already exists (therefore it has already been released). All metadata files are updated based on the remaining versions of the artifact in the repository.
3. The consumer will also check the value of the daysOlder field in the configuration of the repository being scanned. If it is not set to 0 (zero), then the consumer will execute the DaysOldRepositoryPurge. Otherwise, it would execute the RetentionCountRepositoryPurge.
   • DaysOldRepositoryPurge checks when the discovered SNAPSHOT artifact was last modified and if it is older by X (daysOlder value) days then the artifact will be removed from the repository.
   • RetentionCountRepositoryPurge on the other hand, checks if the number of "unique versioned" snapshot artifacts in the directory where the discovered artifact resides is LESS THAN the retentionCount value. If the contents are greater than the retention count, then the oldest snapshot artifact (including associated poms, source jars, javadoc jars, etc.) are removed until the total # of unique versioned artifacts is EQUAL TO the retention count. For example, the discovered artifact is ../artifactX/2.0-SNAPSHOT/artifactX-2.0-SNAPSHOT.jar. RetentionCountRepositoryPurge will get a list of the files in ../artifactX/2.0-SNAPSHOT directory. Lets say, ../artifactX/2.0-SNAPSHOT has the ff. contents: artifactX-2.0-1111111-1.jar, artifactX-2.0-1111111-1.pom, artifactX-2.0-1111100-2.jar, artifactX-2.0-1111100-2.pom, artifactX-2.0-SNAPSHOT.jar and artifactX-2.0-SNAPSHOT.pom. If the retention count is 2, then artifactX-2.0-1111111-1.jar and artifactX-2.0-1111111-1.pom are removed from the repo and the 2 newest artifacts (and its associated files, in this case the poms) are retained.
4. For all these RepositoryPurge implementations, all removed artifacts from the repository are also removed from the database.¹

¹ There is an open issue related to this, please see MRM-455. Aside from this, there is also an open issue regarding the index update after repo purge MRM-454.

Repository Browse

Browse artifacts in the repository...

Repository Search

Search for artifacts from the managed repositories...

Reporting

Repository problems report..

Repository Configuration (Managed and Remote Repository)

Proxy Connectors

Please see Archiva Proxy Connector page.

Network Proxies