Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Blockers, Progress towards next releases, things needing immediate attention, JIRA cleanup

  • 0.6.0 , release blockers reviewed
  • Spark AI summit updates

Pull Request Review 

Unblock reviews, Resolve sticky points

  • All PRs assigned

RFC Reviews

Go over major open ended items

  • One comment Clustering RFC.. 

Open Floor 

new ideas, feedback, suggestions.

  • GenericRecord as the object/row representation in Hudi? `parquet-avro` is painful. 
    • bulk_insert using DataSet... 
    • Schema Evolution   schema on write.. works well for ingestion i.e deltastreamer 
    • Schema on Read.. Datasource/ETL jobs.. 
    • example: id: string ,.. drop id , id : number ... all the old files have id as string..and will have null for old files?  is that ever useful? (schema on read) 
    • example: id: string,  stop publishing id, id_num : number .. on query side, some case/cast and read the data.. (schema on write)
  • Discuss support issues every week? 
    • +1s  Sudha, Me, Nishith, Raymond,.

Major Followups