Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075

Build and test Spark against Scala 2.13

    XMLWordPrintableJSON

Details

    Description

      This umbrella JIRA tracks the requirements for building and testing Spark against the current Scala 2.13 milestone.

      Attachments

        Issue Links

          is a parent of

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-56488 Upgrade Scala 2.13 in 3.5 branch for CVE-2022-36944

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          1.
          		 Use scala-xml explicitly Sub-task Resolved Unassigned
          2.
          		 Remove usage of Traversable Sub-task Closed Sean R. Owen
          3.
          		 Avoid use of Scala collection classes that are removed in 2.13 Sub-task Closed Sean R. Owen
          4.
          		 Remove usage of TraversableOnce Sub-task Resolved Sean R. Owen
          5.
          		 Upgrade breeze to 1.0 Sub-task Closed Sean R. Owen
          6.
          		 Update scalatest, scalacheck, scopt, scala-parser-combinators, clapper classutil Sub-task Closed Sean R. Owen
          7.
          		 Update twitter chill for 2.13 Sub-task Closed Sean R. Owen
          8.
          		 Remove use of procedure-like method definition for 2.13 Sub-task Closed Sean R. Owen
          9.
          		 Fix internal usages of mutable collection for Seq in 2.13 Sub-task Closed Sean R. Owen
          10.
          		 Move scalafmt to Scala 2.12 profile; bump to 0.12 Sub-task Closed Sean R. Owen
          11.
          		 Update Kafka to 2.4.0 for Scala 2.13 Sub-task Closed Jungtaek Lim
          12.
          		 Use scala-parallel-collections library in 2.13 Sub-task Closed Sean R. Owen
          13.
          		 Update REPL for 2.13 Sub-task Closed Karol Chmist
          14.
          		 Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name") Sub-task Closed Sean R. Owen
          15.
          		 Replace ambiguous varargs call parallelize(Array) with parallelize(Seq) Sub-task Closed Sean R. Owen
          16.
          		 Remove use of Unit companion object in favor of () Sub-task Closed Sean R. Owen
          17.
          		 Remove use of TraversableLike Sub-task Closed Sean R. Owen
          18.
          		 Replace Array.deep comparisons with Array.sameElements Sub-task Closed Sean R. Owen
          19.
          		 Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1 Sub-task Closed Dongjoon Hyun
          20.
          		 Replace Ordering.Double with Ordering.Double.TotalOrdering Sub-task Closed Sean R. Owen
          21.
          		 Remove deprecated SparkConf.setAll(Traversable) Sub-task Closed Sean R. Owen
          22.
          		 Use DoubleAsIfIntegral / FloatAsIfIntegral workalike in 2.13 Sub-task Closed Sean R. Owen
          23.
          		 Change classes extending scala collection classes to work with 2.13 Sub-task Closed Sean R. Owen
          24.
          		 No parens for calls to BigDecimal.longValue() etc. Sub-task Closed Sean R. Owen
          25.
          		 Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses Sub-task Closed Unassigned
          26.
          		 Resolve ambiguous overload from RegexParsers in 2.13 Sub-task Closed Sean R. Owen
          27.
          		 Resolve Array + reference type compile problems in 2.13, with sc.parallelize Sub-task Closed Sean R. Owen
          28.
          		 Some imports, function need more explicit resolution in 2.13 Sub-task Closed Sean R. Owen
          29.
          		 Upgrade Genjavadoc to 0.15 Sub-task Closed Kousuke Saruta
          30.
          		 Iterator doesn't refer outer identifier named "iterator" properly in Scala 2.13 Sub-task Closed Kousuke Saruta
          31.
          		 Get tests working for Scala 2.13 build Sub-task Closed Sean R. Owen
          32.
          		 Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility Sub-task Closed Sean R. Owen
          33.
          		 Upgrade Genjavadoc to 0.16 Sub-task Closed Dongjoon Hyun
          34.
          		 Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts Sub-task Closed Dongjoon Hyun
          35.
          		 Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal Sub-task Closed Dongjoon Hyun
          36.
          		 Use HashMap.withDefaultValue in RDDSuite Sub-task Closed Dongjoon Hyun
          37.
          		 Make BlockManagerSuite robust from Scala object size difference Sub-task Closed Dongjoon Hyun
          38.
          		 Update json4s for Scala 2.13 Sub-task Closed Dongjoon Hyun
          39.
          		 Fix TaskSetManagerSuite by hiding `org.apache.spark.FakeSchedulerBackend` Sub-task Closed Dongjoon Hyun
          40.
          		 Fix testCommandAvailable to use POSIX compatible `command -v` Sub-task Closed Hyukjin Kwon
          41.
          		 Fix HistoryServerSuite in Scala 2.13 Sub-task Resolved Unassigned
          42.
          		 Use === instead IndexSeqView Sub-task Closed Dongjoon Hyun
          43.
          		 ResourceAllocator.availableAddrs should be deterministic Sub-task Closed Dongjoon Hyun
          44.
          		 JsonProtocol.accumulablesToJson should be deterministic Sub-task Closed Dongjoon Hyun
          45.
          		 Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource Sub-task Closed Dongjoon Hyun
          46.
          		 Pass all `core` module UTs in Scala 2.13 Sub-task Closed Dongjoon Hyun
          47.
          		 Let sql/catalyst module tests pass for Scala 2.13 Sub-task Closed Yang Jie
          48.
          		 Fix CaseInsensitiveMap in Scala 2.13 Sub-task Closed Karol Chmist
          49.
          		 Pass all `sql/core` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          50.
          		 Ignore deprecation warnings for build with Scala-2.13 and sbt Sub-task Closed Kousuke Saruta
          51.
          		 Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13 Sub-task Closed Kousuke Saruta
          52.
          		 Fix code which causes error when build with sbt and Scala 2.13 Sub-task Closed Kousuke Saruta
          53.
          		 Pass all `sql/hive` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          54.
          		 Pass all `sql/hive-thriftserver` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          55.
          		 Add Scala 2.13 build test in GitHub Action Sub-task Closed Dongjoon Hyun
          56.
          		 Pass all `external/avro` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          57.
          		 Pass all `streaming` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          58.
          		 Pass all `mllib` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          59.
          		 Pass all `mesos` module UTs in Scala 2.13 Sub-task Closed Yang Jie
          60.
          		 Add a Jenkins build and test job for Scala 2.13 Sub-task Resolved Shane Knapp
          61.
          		 Update how to build doc for Scala 2.13 with sbt Sub-task Closed Kousuke Saruta
          62.
          		 Fix SparkBuild.scala to recognize build settings for Scala 2.13 Sub-task Closed Kousuke Saruta
          63.
          		 Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job Sub-task Closed Kousuke Saruta
          64.
          		 Rename classOf[Seq] to classOf[scala.collection.Seq] in PostgresIntegrationSuite for Scala 2.13 Sub-task Closed Kousuke Saruta
          65.
          		 Fix procedure-like declaration compilation warning in Scala 2.13 Sub-task Closed Yang Jie
          66.
          		 Upgrade to Scala 2.13.4 Sub-task Closed Dongjoon Hyun
          67.
          		 Upgrade Scala 2.13 from 2.13.3 to 2.13.4 Sub-task Closed Unassigned
          68.
          		 Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform` Sub-task Closed Dongjoon Hyun
          69.
          		 Suppress unimportant compilation warnings in Scala 2.13 Sub-task Closed Yang Jie
          70.
          		 Word2VecSuite failed for Scala 2.13 Sub-task Closed koert kuipers
          71.
          		 ExpressionEncoderSuite failed in spark-branch-3.1-test-maven-hadoop-*-jdk-*-scala-2.13 Sub-task Closed Yang Jie
          72.
          		 HadoopVersionInfoSuite faild when maven test in Scala 2.13 Sub-task Closed Unassigned
          73.
          		 Java UT failed when testing sql/hive module independently in Scala 2.13 Sub-task Closed Yang Jie
          74.
          		 Add Scala 2.13 packaging and publishing Sub-task Closed Dongjoon Hyun
          75.
          		 Upgrade Scala to avoid Scala's bug about illegal cyclic reference Sub-task Closed Unassigned
          76.
          		 Upgrade Scala to 2.13.5 Sub-task Closed Dongjoon Hyun
          77.
          		 Update sbt version to 1.4.9 Sub-task Closed William Hyun
          78.
          		 Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12 Sub-task Closed Sean R. Owen
          79.
          		 Jekyll fails to generate Scala API docs for Scala 2.13 Sub-task Closed Kousuke Saruta
          80.
          		 Upgrade SBT to 1.5.0 Sub-task Closed Dongjoon Hyun
          81.
          		 Suppress `symbol literal is deprecated` compilation warnings in Scala 2.13 Sub-task Resolved Unassigned
          82.
          		 Upgrade GenJavadoc to 0.17 Sub-task Closed Kousuke Saruta
          83.
          		 Many test cases failed in Scala 2.13 CI Sub-task Closed Takeshi Yamamuro
          84.
          		 Upgrade SBT to 1.5.2 Sub-task Closed Dongjoon Hyun
          85.
          		 Sort elements generated by collect_set in SQLQueryTestSuite Sub-task Closed Takeshi Yamamuro
          86.
          		 Upgrade scalatestplus artifacts to 3.2.9.0 Sub-task Closed Kousuke Saruta
          87.
          		 Re-cleanup `procedure syntax is deprecated` compilation warning in Scala 2.13 Sub-task Closed Yang Jie
          88.
          		 Ensure mllib and kafka-0-10 module can maven test independently in Scala 2.13 Sub-task Closed Yang Jie
          89.
          		 Turn compilation warnings related to `procedure syntax` to errors in Scala 2.13 Sub-task Closed Yang Jie
          90.
          		 Upgrade SBT to 1.5.3 Sub-task Closed Kousuke Saruta
          91.
          		 Upgrade SBT to 1.5.5 Sub-task Closed Kousuke Saruta
          92.
          		 Upgrade scala-maven-plugin to 4.5.3 Sub-task Closed William Hyun
          93.
          		 Upgrade sbt-mima-plugin to 0.9.2 Sub-task Closed Dongjoon Hyun
          94.
          		 Ensure all modules can be maven test independently in Scala 2.13 Sub-task Closed Yang Jie
          95.
          		 Disable MiMa for Scala 2.13 artifacts Sub-task Closed Dongjoon Hyun
          96.
          		 Support Scala 2.13 test in `dev/run-tests.py` Sub-task Closed Dongjoon Hyun
          97.
          		 Upgrade GenJavadoc to 0.18 Sub-task Closed Kousuke Saruta
          98.
          		 Add Scala 2.13 daily build and test GitHub Action job Sub-task Closed Dongjoon Hyun
          99.
          		 SparkContextSuite random failure in Scala 2.13 Sub-task Closed Yang Jie
          100.
          		 Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile Sub-task Closed Lukas Rytz
          101.
          		 Upgrade Scala to 2.13.8 Sub-task Closed Yang Jie

          Activity

            People

              srowen Sean R. Owen
              MasseGuillaume Guillaume Massé
              Sean R. Owen Sean R. Owen
              Votes:
              16 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: