Packaging Dependencies Part 2; and plan on how to

Mapping and packaging dependencies part 1

Hey all, I had my exams during weeks 8 and 9, so I couldn't update my blog nor get much accomplished; but last week was completely free so I managed to finish packaging all the dependencies from packaging dependencies part 1. Since some of you may not remember how I planned to tackle pacakging dependencies I'll mention it here one more time.

I split this task into two sub tasks that can be done independently. The 2 subtasks are as follows:

  • part 1: make the entire project build successfully without :buildSrc:prepare-deps:intellij-sdk:build
    • part 1.1: package these dependencies
  • part 2: package the dependencies in :buildSrc:prepare-deps:intellij-sdk:build; i.e. try to recreate whatever is in it.

This is taken from my last blog which was specifically on packaging dependencies in part 1. Now I am happy to tell all of you that packaging dependencies for part 1 is now complete and all the needed packages are either in the new queue or already in sid archive as of 04 August 2019. I would like to thank ebourg, seamlik and andrewsh for helping me with this.

How to build kotlin 1.3.30 after dependency packaging part 1 and design choices

Before I go into how to build the project as it is now I'll briefly talk of some of the choices I made while packaging dependencies in part 1 and general things you should know.

Two dependencies in part 1 were Jcabi-aether and sonatype-aether, both of these are incompatible with maven-3 and these were only used in one single file in the entire dist task graph. Considering the time it would take to migrate these dependencies to maven-3 I chose to patch out the one file that needed both of these and that change is denoted by this commit. Also it must be noted that so far we are only trying to build the dist task which only and only builds the basic Kotlin compiler; it doesn't build the maven artifacts with poms nor does it build the kotlin-gradle-plugin. Those things are built and installed in the local maven repository (.m2 file in surce project when you invoke debuild) using the install task which I am planning to do once we finish successfully building the dist task. Invoking the install task in our master as of Aug 04, 2019 will build and install all available maven artifacts into the local maven repo but this again will not have kotlin-gradle-plugin or such since I have removed those subprojects as they aren't needed by the dist task. Keeping them would mean that I have to convert and patch them to groovy if they are written in .kts since they are evaluated during the initialization phase.

Now we are ready to build the project. I have written a simple makefile which copies all the needed bootstrap jars and prebuilts to their proper places. All you need to build the project is:

1.git clone https://salsa.debian.org/m36-guest/kotlin-1.3.30.git
2.cd kotlin-1.3.30
3.git checkout buildv1
4.debian/pseudoBootstrap bootstrap
5.debuild -b -rfakeroot -us -uc
Note that we need only do steps 1 though 4 the very first time you are building this project. everytime after that just invoke step 5

Packaging dependencies part 2

Now packaging dependencies part 2 involves package the dependencies in :buildSrc:prepare-deps:intellij-sdk:build. This is the folder that is taking up the most space in Kotlin-1.3.30-temp-requirements. The sole purpose of this task is reduce the jars in this folder and substitute them with JAR from the Debian environment. I have managed to map out the needed jars from these for the dist task graph and they are

saif@Hope:/srv/chroot/KotlinCh/home/kotlin/kotlin-1.3.30-debian-maintained/buildSrc/prepare-deps/intellij-sdk/repo/kotlin.build.custom.deps/183.5153.4$ ls -R
.:
intellij-core  intellij-core.ivy.xml  intellijUltimate  intellijUltimate.ivy.xml  jps-standalone  jps-standalone.ivy.xml

./intellij-core:
asm-all-7.0.jar  intellij-core.jar  java-compatibility-1.0.1.jar

./intellijUltimate:
lib

./intellijUltimate/lib:
asm-all-7.0.jar  guava-25.1-jre.jar  jna.jar           log4j.jar      openapi.jar    picocontainer-1.2.jar  platform-impl.jar   trove4j.jar
extensions.jar   jdom.jar            jna-platform.jar  lz4-1.3.0.jar  oro-2.0.8.jar  platform-api.jar       streamex-0.6.7.jar  util.jar

./jps-standalone:
jps-model.jar

This folder is treated as an ant repository and the code to that is here. Build.gradle files use this via methods like this which tells the project to take only the needed jars from the collection. I am planning on replacing this with just plain old maven repository resolution using format like compile(groupID:artifactId:version) but we will need the jars to be in our system anyways, atleast now we know that this particular file structure can be avoided.

Please note that these jars listed above by me are only needed for the dist task and the ones needed for other subprojects in the original install task can still be found here.

The following are the dependencies need for part 2. * denotes what I am not sure of. Contact me before you attempt to pacakge any of the intellij dependencies as we only need parts from those and I have a script to tell what we need.

  1. → java-compatibility-1.0.1 → https://github.com/JetBrains/intellij-deps-java-compatibility (DONE: here)
  2. → jps-model → https://github.com/JetBrains/intellij-community/tree/master/jps (DONE: here)
  3. → intellij-core → https://github.com/JetBrains/intellij-community/tree/183.5153 (DONE: here)
  4. → streamex-0.6.7 → https://github.com/amaembo/streamex/tree/streamex-0.6.7 (DONE: here)
  5. → guava-25.1 → https://github.com/google/guava/tree/v25.1 (DONE: Used guava-19 from libguava-java)
  6. → lz4-java → https://github.com/lz4/lz4-java/blob/1.3.0/build.xml (DONE:here)
  7. → libjna-java & libjna-platform-java recompiled in jdk 8. → https://salsa.debian.org/java-team/libjna-java (DONE: commit)
  8. → liboro-java recompiled in jdk8 → https://salsa.debian.org/java-team/liboro-java (DONE: commit)
  9. → picocontainer-1.3 refining → https://salsa.debian.org/java-team/libpicocontainer-1-java (DONE: here)
  10. → platform-api → https://github.com/JetBrains/intellij-community/tree/183.5153/platform (DONE: here)
  11. → util → https://github.com/JetBrains/intellij-community/tree/183.5153/platform (DONE: here)
  12. → platform-impl → https://github.com/JetBrains/intellij-community/tree/183.5153/platform (DONE: here)
  13. → extensions → https://github.com/JetBrains/intellij-community/tree/183.5153/platform (DONE: here)

So if any of you want to help please kindly take on any of these and package them.

NOTE — ping me if you want to build Kotlin in your system and are stuck!

Here is a link to the work I have done so far. You can find me as m36 or m36[m] on #debian-mobile and #debian-java on OFTC.

I’ll try to maintain this blog and post the major updates weekly.

Tweet Share