Versions Compared

Old Version 17

changes.mady.by.user Balaji Varadarajan

Saved on

compared with

New Version Current

changes.mady.by.user Udit Mehrotra

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The idea here is to implement a new Spark Relation and Spark RDD that will be used for scanning and reading bootstrapped tables. The custom relation will implement PruneFilteredScan to allow for supporting filters pushdown and column pruning. For the RDD, each partition will be data file + optional skeleton file combination which will be sent to one task to perform the merge and return the results.

...