Incorrect URL generated by MavenPublishArtifactManager in azure snapshot builds

[jazminebarroga]

Summary

I wanted to distribute a snapshot build via Cocoapods but the url created by the MavenPublishArtifactManager.artifactPath is not the proper url to the snapshot build.

Details

We're using azure as our artifact repository for xcframeworks and one of its limitations is that you cannot overwrite published artifacts unless you append SNAPSHOT to its version. Snapshot builds have a different naming convention and do not follow the ...$kmmbridgeArtifactId-$version.zip url naming that the MavenPublishArtifactManager.artifactPath creates.

I also previously created a custom gradle task to make sure that the actual version in the generated podspec follows the proper naming convention so it passes pod lint such that if the version is 0.1.0-SNAPSHOT it becomes 0.1.0.beta in the podspec when generated.

Reproduction

1. Change artifact version to 0.1.0-SNAPSHOT
2. Use azure as artifact repository
3. Run kmmBridgePublish

Expected result

The generated podspec should refer to the actual URL of the maven artifact which is something like this ...sdk-kmmbridge/0.1.0-SNAPSHOT/sdk-kmmbridge-0.1.0-20230418.083616-1.zip

Current state

The generated podspec refers to ...sdk-kmmbridge/0.1.0-SNAPSHOT/sdk-kmmbridge-0.1.0-SNAPSHOT.zip which is not the actual URL of the maven artifact

[russhwolf]

We don't have any explicit snapshot support at the moment. I'm not sure I fully understand the azure issue, but you might be able to work around by defining a custom version writer like this:

class SuffixedVersionWriter(private val suffix: String, private val delegate: VersionWriter): VersionWriter by delegate {
    override fun scanVersions(project: Project, block: (Sequence<String>) -> Unit) {
        delegate.scanVersions(project) { sequence ->  
            block(sequence.map { it.removeSuffix(suffix) })
        }
    }

    override fun writeMarkerVersion(project: Project, version: String) {
        delegate.writeMarkerVersion(project, version + suffix)
    }

    override fun writeFinalVersion(project: Project, version: String) {
        delegate.writeFinalVersion(project, version + suffix)
    }
}

and then in your kmmbridge block do

versionWriter.set(SuffixedVersionWriter("-SNAPSHOT", GitRemoteVersionWriter()) // delegate to whatever version writer is currently being used

but I haven't tested that so it might not quite work as-is.

[jazminebarroga]

Let me provide more details. So for example, using kmmbridge, I am already able to upload the xcframework in an azure maven repository with the version 0.1.0-SNAPSHOT and so I am able to generate a podspec that looks like this

Pod::Spec.new do |spec|
    spec.name                     = 'SDK'
    spec.version                  = '0.1.0.beta'
    spec.homepage                 = 'https://www.google.com'
    spec.source                   = { 
                                      :http => 'https://pkgs.dev.azure.com/[redacted]/sdk-kmmbridge/0.1.0-SNAPSHOT/sdk-kmmbridge-0.1.0-SNAPSHOT.zip',
                                      :type => 'zip',
                                      :headers => ['Accept: application/octet-stream']
                                    }
    spec.authors                  = ''
    spec.license                  = ''
    spec.summary                  = 'API'
    spec.vendored_frameworks      = 'sdk.xcframework'
    spec.libraries                = 'c++'
    spec.ios.deployment_target = '14'
           
end

The url however is incorrect in the generated podspec, because the url for snapshot versions is actually like this:

https://pkgs.dev.azure.com/[redacted]/sdk-kmmbridge/0.1.0-SNAPSHOT/sdk-kmmbridge-0.1.0-20230420.015940-1.zip

The upload date is appended as suffix.

[russhwolf]

Can you show your kmmbridge gradle configuration? I want to understand how you're setting the url

[jazminebarroga]

Here

/**
 * Publishing configurations
 */
def localProperties = new Properties()
try {
    localProperties.load(project.rootProject.file("local.properties").newDataInputStream())
} catch (FileNotFoundException e) {
    println("Local properties file not found. $e")
}

def publishingGroupId = "group id"
def publishingVersion = localProperties.getProperty("publishing.version") ?: "0.0.0"
def azurePublishingUrl = localProperties.getProperty("azure.url")
def azurePublishingUser = localProperties.getProperty("azure.user")
def azurePublishingPw = localProperties.getProperty("azure.pw")
def cocoapodsSummary = localProperties.getProperty("cocoapods.summary") ?: ""
def cocoapodsHomepage = localProperties.getProperty("cocoapods.homepage") ?: ""

subprojects { project ->

    apply plugin: "maven-publish"
    apply plugin: libs.plugins.kmmbridge.get().pluginId
    apply plugin: "org.jetbrains.kotlin.native.cocoapods"
    apply plugin: "org.jetbrains.kotlin.multiplatform"

    project.group = publishingGroupId
    project.version = publishingVersion

    publishing {
        repositories {
            if (azurePublishingUrl != null && !azurePublishingUrl.isEmpty()) {
                maven {
                    name = "azure"
                    url = uri(azurePublishingUrl)
                    credentials {
                        username = azurePublishingUser
                        password = azurePublishingPw
                    }
                }
            }
        }
    }

    kmmbridge {
        mavenPublishArtifacts(project, null, null)
        manualVersions()

        tasks.register("overwriteKmmbridgeVersionFile", OverwriteKmmbridgeVersionFileTask) {
            buildDirectory = getBuildDir()
        }

        cocoapods(project, "specs url", true, false)

        afterEvaluate {
            tasks.overwriteKmmbridgeVersionFile.mustRunAfter tasks.uploadXCFramework
            tasks.generateReleasePodspec.dependsOn tasks.overwriteKmmbridgeVersionFile
        }
    }

    kotlin {
        cocoapods {
            summary = cocoapodsSummary
            homepage = cocoapodsHomepage
            name = "name"
            ios.deploymentTarget = "14"
        }
    }
}

abstract class OverwriteKmmbridgeVersionFileTask extends DefaultTask {

    @Input
    abstract Property<File> getBuildDirectory()

    @TaskAction
    def execute() {

        File versionFile = new File(buildDirectory.get().toString() + "/faktory/version")
        String newVersion = versionFile.readLines().first().replaceAll("-SNAPSHOT", ".beta")
        versionFile.write(newVersion)
    }
}

[arcadii-meister]

I have encountered the same issue with snapshot versions, but I am using Google Cloud Artifact Registry. However, it being a Maven repository, it is the same limitation.

I could overcome it by using this code, thank you @jazminebarroga for inspiration:

kmmbridge {
    mavenPublishArtifacts()
    manualVersions()
    noGitOperations()
    spm(useCustomPackageFile = true)

    val version = version.toString()

    if (!version.contains("-SNAPSHOT")) {
        return@kmmbridge
    }

    val fixKMMBridgeSnapshotVersion by tasks.registering {
        group = "kmmbridge"

        doLast {
            val snapShotVersion = project.dependencies
                .create(project.group.toString(), "${frameworkName.get()}-kmmbridge", version)
                .let { configurations.detachedConfiguration(it) }
                .apply { resolutionStrategy.cacheChangingModulesFor(0, TimeUnit.MINUTES) }
                .resolvedConfiguration.resolvedArtifacts.firstOrNull()
                ?.id?.componentIdentifier?.let { it as? MavenUniqueSnapshotComponentIdentifier }
                ?.timestamp
                ?: error("Cannot resolve component timestamp")

            with(file("$buildDir/faktory/url")) {
                readText()
                    .replace("SNAPSHOT.zip", "$snapShotVersion.zip")
                    .let(::writeText)
            }
        }
    }

    /**
     * Adds the snapshot fix task to the KMMBridge's task chain.
     */
    afterEvaluate {
        val uploadXCFramework by tasks.existing
        fixKMMBridgeSnapshotVersion.dependsOn(uploadXCFramework)

        val updatePackageSwift by tasks.getting
        updatePackageSwift.dependsOn(fixKMMBridgeSnapshotVersion)
        
        // Forces the task always rerun to write the new version to the package file.
        // Otherwise, it is cached and Package.swift is not updated.
        updatePackageSwift.outputs.upToDateWhen { false }
    }
}

The idea is to use the Maven snapshot resolution strategy to get the latest timestamp and replace -SNAPSHOT with it, so Package.swift will have a link to the specific file.

// BEGIN KMMBRIDGE VARIABLES BLOCK (do not edit)
let remoteKotlinUrl = "https://europe-maven.pkg.dev/.../common-kmmbridge/0.1.1-SNAPSHOT/common-kmmbridge-0.1.1-20230725.122208-80.zip"
let remoteKotlinChecksum = "..."
let packageName = "common"
// END KMMBRIDGE BLOCK

Replacing -SNAPSHOT only for "$buildDir/faktory/url" file is intentional as we need to fix versioning only for XCFramework, but we can leave it as SNAPSHOT for Android/Java since they are using Maven to resolve it when consuming.

At first, I wanted to try just to parse a Maven metadata file from a remote repository, but I decided to use Gradle API to reuse the already provided authentication to the repository.

I have also tried to use createArtifactResolutionQuery(), but I couldn't bypass the local cache, so after each uploadXCFramework, that created a new SNAPSHOT file, createArtifactResolutionQuery() was returning cached artifacts.

[kpgalligan]

We do indeed "guess" the maven url. The code above is interesting. I'd have to think through how this might impact SPM config, as there's a global "version" assumption in the code to some degree, and SPM can't deal well with non-semver versions, but it may not be a big deal.

[arcadii-meister]

SPM can't deal well with non-semver versions

@kpgalligan Exactly, because of the SPM semver requirements I have to tag my commits with versions excluding -SNAPSHOT postfix. So for the iOS devs, it looks like a regular version where only the commit revision is changing in the Package.resolved file.

Turns out Xcode doesn't like it very much and iOS devs need to constantly Reset Package Caches and sometimes clear the derived data folder, which in our case takes quite some time and is very annoying.
Because of that, I am considering avoiding SNAPSHOT and just bumping up the version for each new commit. However, it looks like it is the SPM issue and it is not connected only to the KMP artifacts.

[kpgalligan]

Revisiting this, as KMMBridge is getting a fairly extensive refactor.

The maven publishing works, but is rather ugly as it's a bit of a hack. However, it is also rather useful. I tend to avoid SNAPSHOT builds in general because I've run into issue with Gradle not actually pulling the latest, then you're chasing ghosts. Not with iOS/kmmbridge builds. JVM builds. In any case, adding formal support for this makes sense.

On SPM and semver, we're about to launch a new update. Not just to KMMBridge, but the ability to locally debug Kotlin from published SPM builds. That changes some of my thinking on dev workflows. It may not be of use to everybody, but for dev builds, pointing SPM at a branch instead of a version makes publishing and testing a lot simpler, depending on the use case. I did spend some time trying to get valid semver "pre-release" builds working, but Xcode/SPM essentially ignore that, and in any case, if you're expecting to be able to "see" some changes and not others in an SPM build, which would be the expectation in a dev scenario where the different teams are working on a specific feature, the branch approach makes more sense.

Overall, a lot of these difficulties stem from trying to use published library builds in a workflow that really should be directly managed with git and source code. It's trying to use library publication in a dev situation where there are frequent, and potentially conflicting, updates. That, of course, isn't every scenario, but many that I run into when chatting with teams.

I've published much about it, and will certainly publish more: https://touchlab.co/kmp-teams-piloting-vs-scaling