Maven trying to download all jars from my private S3 repo

Build Environment
1

I have a private repository in S3 that I use to release a library that then I use in another project.

To release, I have this setup:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.7.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>

    <extensions>
        <extension>
            <groupId>com.allogy.maven.wagon</groupId>
            <artifactId>maven-s3-wagon</artifactId>
            <version>1.1.0</version>
        </extension>
    </extensions>
</build>

<distributionManagement>
    <repository>
        <id>projectx-aws-release</id>
        <name>ProjectX AWS Release Repository</name>
        <url>s3://projectx-support/maven2/release</url>
    </repository>
    <snapshotRepository>
        <id>projectx-aws-snapshots</id>
        <name>Projectx AWS Snapshot Repository</name>
        <url>s3://projectx-support/maven2/snapshots</url>
    </snapshotRepository>
</distributionManagement>

and then, on the application that’s using that library:

<build>
    ...

    <extensions>
        <extension>
            <groupId>com.allogy.maven.wagon</groupId>
            <artifactId>maven-s3-wagon</artifactId>
            <version>1.1.0</version>
        </extension>
    </extensions>
</build>

<repositories>
    <repository>
        <id>projectX-aws-release</id>
        <url>s3://projectX-support/maven2/release</url>
    </repository>
    <repository>
        <id>projectX-aws-snapshots</id>
        <url>s3://projectX-support/maven2/snapshots</url>
    </repository>
</repositories>

When running mvn -s .circleci/settings.xml dependency:go-offline something odd happens. It quickly downloads all the dependencies from Maven Central:

mvn -s .circleci/settings.xml dependency:go-offline
[INFO] Scanning for projects...
Downloading: http://repo.maven.apache.org/maven2/com/allogy/maven/wagon/maven-s3-wagon/1.1.0/maven-s3-wagon-1.1.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/allogy/maven/wagon/maven-s3-wagon/1.1.0/maven-s3-wagon-1.1.0.pom (7 KB at 56.6 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/allogy/allogy-maven-pom-oss/2.0/allogy-maven-pom-oss-2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/allogy/allogy-maven-pom-oss/2.0/allogy-maven-pom-oss-2.0.pom (10 KB at 970.1 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/google/guava/guava/15.0/guava-15.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/google/guava/guava/15.0/guava-15.0.pom (7 KB at 678.4 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/google/guava/guava-parent/15.0/guava-parent-15.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/google/guava/guava-parent/15.0/guava-parent-15.0.pom (8 KB at 895.3 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/7/oss-parent-7.pom
.........

but then it seems to also attempt to download them from my own private repo, which clearly doesn’t have them:

[INFO] Plugin Resolved: copy-rename-maven-plugin-1.0.1.jar
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 17ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar (86 KB at 9475.0 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 10ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar (119 KB at 11892.6 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 10ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar (156 KB at 12990.7 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 9ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.jar (13 KB at 2098.3 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-artifact/2.0.9/maven-artifact-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-artifact/2.0.9/maven-artifact-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 12ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-artifact/2.0.9/maven-artifact-2.0.9.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.0.9/maven-artifact-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.0.9/maven-artifact-2.0.9.jar (87 KB at 10853.1 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/codehaus/plexus/plexus-utils/1.5.8/plexus-utils-1.5.8.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/codehaus/plexus/plexus-utils/1.5.8/plexus-utils-1.5.8.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 14ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/codehaus/plexus/plexus-utils/1.5.8/plexus-utils-1.5.8.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.5.8/plexus-utils-1.5.8.jar
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.5.8/plexus-utils-1.5.8.jar (262 KB at 20121.2 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 37ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
Downloaded: http://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar (9 KB at 1035.4 KB/sec)
[INFO]     Plugin Dependency Resolved: maven-model-2.0.9.jar
[INFO]     Plugin Dependency Resolved: maven-project-2.0.9.jar
[INFO]     Plugin Dependency Resolved: maven-core-2.0.9.jar
[INFO]     Plugin Dependency Resolved: maven-plugin-api-2.0.9.jar
[INFO]     Plugin Dependency Resolved: maven-artifact-2.0.9.jar
[INFO]     Plugin Dependency Resolved: plexus-utils-1.5.8.jar
[INFO]     Plugin Dependency Resolved: plexus-build-api-0.0.7.jar
[INFO] Plugin Resolved: maven-deploy-plugin-2.7.jar
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 12ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar (13 KB at 1794.1 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 10ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: s3://projectx-support/maven2/snapshots/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
[WARN] s3://projectx-support/maven2/snapshots - Connection refused
[INFO] Logged off - projectx-support
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar (114 KB at 11346.0 KB/sec)
Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/release/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 9ms Amount: 0 bytes Throughput: 0.000 bytes/s

In my own computer this takes a few seconds, but on CircleCI it’s extremely slow (not sure why). What’s going on? Why is it doing that? How do I stop it?

2

Can you clarify something? The log samples your share have different artifacts, so it’s not actually trying to download the same things, correct? Maven downloads a few core maven internals first, always from central unless a mirror is specified. It will then use those dependencies to begin parsing plugins and dependencies of your specific project. I think that’s why you see a bunch to central first.

As for the ordering once it gets to project specific dependencies, the project pom will always come before the settings.xml, so it follows that maven will try to download from your s3 bucket first, then fallback to central. You can try specifying maven central in your pom as well if you want it to try their first (I think counterintuitively in the reverse order to try them).

Hopefully that explains some of the ordering, but I could only speculate on the performance differences running locally. Verifying maven versions and output from effective-pom might be a decent sanity check.

Also, it might be worth using CircleCI’s caching features for maven to avoid the time of repetitive downloads.

https://circleci.com/docs/2.0/caching/

3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.