Commits

Anonymous committed 883c34b

Reenable ALTER TABLE DROP with new semantics

patch by Aleksey Yeschenko; reviewed by Jonathan Ellis for
CASSANDRA-3919

  • Participants
  • Parent commits 7a6fbc1

Comments (0)

Files changed (9)

     - Disabling autocompactions by setting min/max compaction threshold to 0
       has been deprecated, instead, use the nodetool commands 'disableautocompaction'
       and 'enableautocompaction' or set the compaction strategy option enabled = false
+    - ALTER TABLE DROP has been reenabled for CQL3 tables and has new semantics now.
+      See https://cassandra.apache.org/doc/cql3/CQL.html#alterTableStmt and
+      https://issues.apache.org/jira/browse/CASSANDRA-3919 for details.
 
 
 1.2.4

File doc/cql3/CQL.textile

 
 <instruction> ::= ALTER <identifier> TYPE <type>
                 | ADD   <identifier> <type>
+                | DROP  <identifier>
                 | WITH  <option> ( AND <option> )*
 p. 
 __Sample:__
 The @<tablename>@ is the table name optionally preceded by the keyspace name.  The @<instruction>@ defines the alteration to perform:
 * @ALTER@: Update the type of a given defined column. Note that the type of the "clustering keys":#createTablepartitionClustering cannot be modified as it induces the on-disk ordering of rows. Columns on which a "secondary index":#createIndexStmt is defined have the same restriction. Other columns are free from those restrictions (no validation of existing data is performed), but it is usually a bad idea to change the type to a non-compatible one, unless no data have been inserted for that column yet, as this could confuse CQL drivers/tools.
 * @ADD@: Adds a new column to the table. The @<identifier>@ for the new column must not conflict with an existing column. Moreover, columns cannot be added to tables defined with the @COMPACT STORAGE@ option.
+* @DROP@: Removes a column from the table. Dropped columns will immediately become unavailable in the queries and will not be included in compacted sstables in the future. If a column is readded, queries won't return values written before the column was last dropped. It is assumed that timestamps represent actual time, so if this is not your case, you should NOT readd previously dropped columns. Columns can't be dropped from tables defined with the @COMPACT STORAGE@ option.
 * @WITH@: Allows to update the options of the table. The "supported @<option>@":#createTableOptions (and syntax) are the same as for the @CREATE TABLE@ statement except that @COMPACT STORAGE@ is not supported. Note that setting any @compaction@ sub-options has the effect of erasing all previous @compaction@ options, so you  need to re-specify all the sub-options if you want to keep them. The same note applies to the set of @compression@ sub-options.
 
-Dropping a column is no yet supported but is on "the roadmap":https://issues.apache.org/jira/browse/CASSANDRA-3919. In the meantime, a declared but unused column has no impact on performance nor uses any storage.
-
 h3(#dropTableStmt). DROP TABLE
 
 __Syntax:__
 
 The following describes the addition/changes brought for each version of CQL.
 
+h3. 3.1.0
+
+* "ALTER TABLE":#alterTableStmt @DROP@ option has been reenabled for CQL3 tables and has new semantics now: the space formerly used by dropped columns will now be eventually reclaimed (post-compaction). You should not readd previously dropped columns unless you use timestamps with microsecond precision (see "CASSANDRA-3919":https://issues.apache.org/jira/browse/CASSANDRA-3919 for more details).
+
 h3. 3.0.2
 
 * Type validation for the "constants":#constants has been fixed. For instance, the implementation used to allow @'2'@ as a valid value for an @int@ column (interpreting it has the equivalent of @2@), or @42@ as a valid @blob@ value (in which case @42@ was interpreted as an hexadecimal representation of the blob). This is no longer the case, type validation of constants is now more strict. See the "data types":#types section for details on which constant is allowed for which type.

File pylib/cqlshlib/helptopics.py

         Currently, COUNT is the only function supported by CQL.
         """
 
+    def help_alter_drop(self):
+        print """
+        ALTER TABLE: dropping a typed column
+
+          ALTER TABLE addamsFamily DROP gender;
+
+        An ALTER TABLE ... DROP statement removes the type of a column
+        from the column family metadata. Dropped columns will immediately
+        become unavailable in the queries and will not be included in
+        compacted sstables in the future. If a column is readded, queries
+        won't return values written before the column was last dropped.
+        It is assumed that timestamps represent actual time, so if this
+        is not your case, you should NOT readd previously dropped columns.
+        Columns can't be dropped from tables defined with COMPACT STORAGE.
+        """
+
     def help_create(self):
         super(CQL3HelpTopics, self).help_create()
         print "          HELP CREATE_USER;"

File src/java/org/apache/cassandra/config/CFMetaData.java

 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import org.apache.cassandra.cql3.ColumnNameBuilder;
 import org.apache.cassandra.cql3.CFDefinition;
 import org.apache.cassandra.cql3.QueryProcessor;
 import org.apache.cassandra.cql3.UntypedResultSet;
                                                                   + "strategy_class text,"
                                                                   + "strategy_options text"
                                                                   + ") WITH COMPACT STORAGE AND COMMENT='keyspace definitions' AND gc_grace_seconds=8640");
+
     public static final CFMetaData SchemaColumnFamiliesCf = compile(9, "CREATE TABLE " + SystemTable.SCHEMA_COLUMNFAMILIES_CF + "("
                                                                        + "keyspace_name text,"
                                                                        + "columnfamily_name text,"
                                                                        + "default_write_consistency text,"
                                                                        + "speculative_retry text,"
                                                                        + "populate_io_cache_on_flush boolean,"
+                                                                       + "dropped_columns map<text, bigint>,"
                                                                        + "PRIMARY KEY (keyspace_name, columnfamily_name)"
                                                                        + ") WITH COMMENT='ColumnFamily definitions' AND gc_grace_seconds=8640");
+
     public static final CFMetaData SchemaColumnsCf = compile(10, "CREATE TABLE " + SystemTable.SCHEMA_COLUMNS_CF + "("
                                                                + "keyspace_name text,"
                                                                + "columnfamily_name text,"
     private volatile int defaultTimeToLive = DEFAULT_DEFAULT_TIME_TO_LIVE;
     private volatile SpeculativeRetry speculativeRetry = DEFAULT_SPECULATIVE_RETRY;
     private volatile boolean populateIoCacheOnFlush = DEFAULT_POPULATE_IO_CACHE_ON_FLUSH;
+    private volatile Map<ByteBuffer, Long> droppedColumns = new HashMap<ByteBuffer, Long>();
 
     /*
      * All CQL3 columns definition are stored in the column_metadata map.
     public CFMetaData defaultTimeToLive(int prop) {defaultTimeToLive = prop; return this;}
     public CFMetaData speculativeRetry(SpeculativeRetry prop) {speculativeRetry = prop; return this;}
     public CFMetaData populateIoCacheOnFlush(boolean prop) {populateIoCacheOnFlush = prop; return this;}
+    public CFMetaData droppedColumns(Map<ByteBuffer, Long> cols) {droppedColumns = cols; return this;}
 
     public CFMetaData(String keyspace, String name, ColumnFamilyType type, AbstractType<?> comp, AbstractType<?> subcc)
     {
         return UUID.nameUUIDFromBytes(ArrayUtils.addAll(ksName.getBytes(), cfName.getBytes()));
     }
 
-    private void init()
-    {
-        updateCfDef(); // init cqlCfDef
-    }
-
     private static CFMetaData newSystemMetadata(String keyspace, String cfName, int oldCfId, String comment, AbstractType<?> comparator, AbstractType<?> subcc)
     {
         ColumnFamilyType type = subcc == null ? ColumnFamilyType.Standard : ColumnFamilyType.Super;
                       .indexInterval(oldCFMD.indexInterval)
                       .speculativeRetry(oldCFMD.speculativeRetry)
                       .memtableFlushPeriod(oldCFMD.memtableFlushPeriod)
-                      .populateIoCacheOnFlush(oldCFMD.populateIoCacheOnFlush);
+                      .populateIoCacheOnFlush(oldCFMD.populateIoCacheOnFlush)
+                      .droppedColumns(oldCFMD.droppedColumns);
     }
 
     /**
         return defaultTimeToLive;
     }
 
+    public Map<ByteBuffer, Long> getDroppedColumns()
+    {
+        return droppedColumns;
+    }
+
     public boolean equals(Object obj)
     {
         if (obj == this)
             .append(indexInterval, rhs.indexInterval)
             .append(speculativeRetry, rhs.speculativeRetry)
             .append(populateIoCacheOnFlush, rhs.populateIoCacheOnFlush)
+            .append(droppedColumns, rhs.droppedColumns)
             .isEquals();
     }
 
             .append(indexInterval)
             .append(speculativeRetry)
             .append(populateIoCacheOnFlush)
+            .append(droppedColumns)
             .toHashCode();
     }
 
         speculativeRetry = cfm.speculativeRetry;
         populateIoCacheOnFlush = cfm.populateIoCacheOnFlush;
 
+        if (!cfm.droppedColumns.isEmpty())
+            droppedColumns = cfm.droppedColumns;
+
         MapDifference<ByteBuffer, ColumnDefinition> columnDiff = Maps.difference(column_metadata, cfm.column_metadata);
         // columns that are no longer needed
         for (ColumnDefinition cd : columnDiff.entriesOnlyOnLeft().values())
         cf.addColumn(DeletedColumn.create(ldt, timestamp, cfName, "compaction_strategy_options"));
         cf.addColumn(DeletedColumn.create(ldt, timestamp, cfName, "index_interval"));
 
+        for (Map.Entry<ByteBuffer, Long> entry : droppedColumns.entrySet())
+            cf.addColumn(new DeletedColumn(makeDroppedColumnName(entry.getKey()), ldt, timestamp));
+
         for (ColumnDefinition cd : column_metadata.values())
             cd.deleteFromSchema(rm, cfName, getColumnDefinitionComparator(cd), timestamp);
 
         cf.addColumn(Column.create(indexInterval, timestamp, cfName, "index_interval"));
         cf.addColumn(Column.create(speculativeRetry.toString(), timestamp, cfName, "speculative_retry"));
 
+        for (Map.Entry<ByteBuffer, Long> entry : droppedColumns.entrySet())
+            cf.addColumn(new Column(makeDroppedColumnName(entry.getKey()), LongType.instance.decompose(entry.getValue()), timestamp));
+
         // Save the CQL3 metadata "the old way" for compatibility sake
         cf.addColumn(Column.create(aliasesToJson(partitionKeyColumns), timestamp, cfName, "key_aliases"));
         cf.addColumn(Column.create(aliasesToJson(clusteringKeyColumns), timestamp, cfName, "column_aliases"));
             if (result.has("value_alias"))
                 cfm.addColumnMetadataFromAliases(Collections.<ByteBuffer>singletonList(result.getBytes("value_alias")), cfm.defaultValidator, ColumnDefinition.Type.COMPACT_VALUE);
 
+            if (result.has("dropped_columns"))
+                cfm.droppedColumns(convertDroppedColumns(result.getMap("dropped_columns", UTF8Type.instance, LongType.instance)));
+
             return cfm;
         }
         catch (SyntaxException e)
         return rawAliases;
     }
 
+    private static Map<ByteBuffer, Long> convertDroppedColumns(Map<String, Long> raw)
+    {
+        Map<ByteBuffer, Long> converted = Maps.newHashMap();
+        for (Map.Entry<String, Long> entry : raw.entrySet())
+            converted.put(UTF8Type.instance.decompose(entry.getKey()), entry.getValue());
+        return converted;
+    }
+
+    private ByteBuffer makeDroppedColumnName(ByteBuffer column)
+    {
+        ColumnNameBuilder builder = SchemaColumnFamiliesCf.cqlCfDef.getColumnNameBuilder();
+        builder.add(UTF8Type.instance.decompose(cfName));
+        builder.add(UTF8Type.instance.decompose("dropped_columns"));
+        return builder.add(column).build();
+    }
+
     /**
      * Convert current metadata into schema mutation
      *
         return removed;
     }
 
+    public void recordColumnDrop(ColumnDefinition def)
+    {
+        assert def.componentIndex != null;
+        droppedColumns.put(def.name, FBUtilities.timestampMicros());
+    }
+
     public void renameColumn(ByteBuffer from, String strFrom, ByteBuffer to, String strTo) throws InvalidRequestException
     {
         ColumnDefinition def = column_metadata.get(from);
             .append("speculative_retry", speculativeRetry)
             .append("indexInterval", indexInterval)
             .append("populateIoCacheOnFlush", populateIoCacheOnFlush)
+            .append("droppedColumns", droppedColumns)
             .toString();
     }
 }

File src/java/org/apache/cassandra/cql3/Cql.g

     : K_ALTER K_COLUMNFAMILY cf=columnFamilyName
           ( K_ALTER id=cident K_TYPE v=comparatorType { type = AlterTableStatement.Type.ALTER; }
           | K_ADD   id=cident v=comparatorType        { type = AlterTableStatement.Type.ADD; }
-          // | K_DROP  id=cident                         { type = AlterTableStatement.Type.DROP; }
+          | K_DROP  id=cident                         { type = AlterTableStatement.Type.DROP; }
           | K_WITH  properties[props]                 { type = AlterTableStatement.Type.OPTS; }
           | K_RENAME                                  { type = AlterTableStatement.Type.RENAME; }
                id1=cident K_TO toId1=cident { renames.put(id1, toId1); }

File src/java/org/apache/cassandra/cql3/QueryProcessor.java

 
 public class QueryProcessor
 {
-    public static final SemanticVersion CQL_VERSION = new SemanticVersion("3.0.1");
+    public static final SemanticVersion CQL_VERSION = new SemanticVersion("3.1.0");
 
     private static final Logger logger = LoggerFactory.getLogger(QueryProcessor.class);
 

File src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java

             case DROP:
                 if (cfDef.isCompact)
                     throw new InvalidRequestException("Cannot drop columns from a compact CF");
+                if (!cfDef.isComposite)
+                    throw new InvalidRequestException("Cannot drop columns from a non-CQL3 CF");
                 if (name == null)
                     throw new InvalidRequestException(String.format("Column %s was not found in table %s", columnName, columnFamily()));
 
                         }
                         assert toDelete != null;
                         cfm.removeColumnDefinition(toDelete);
+                        cfm.recordColumnDrop(toDelete);
                         break;
                 }
                 break;

File src/java/org/apache/cassandra/db/ColumnFamilyStore.java

 import org.apache.cassandra.db.index.SecondaryIndex;
 import org.apache.cassandra.db.index.SecondaryIndexManager;
 import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.CompositeType;
 import org.apache.cassandra.dht.*;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.exceptions.ConfigurationException;
             // remove columns if
             // (a) the column itself is gcable or
             // (b) the column is shadowed by a CF tombstone
-            if (c.getLocalDeletionTime() < gcBefore || cf.deletionInfo().isDeleted(c))
+            // (c) the column has been dropped from the CF schema (CQL3 tables only)
+            if (c.getLocalDeletionTime() < gcBefore || cf.deletionInfo().isDeleted(c) || isDroppedColumn(c, cf.metadata()))
             {
                 iter.remove();
                 indexer.remove(c);
         removeDeletedColumnsOnly(cf, gcBefore, SecondaryIndexManager.nullUpdater);
     }
 
+    // returns true if
+    // 1. this column has been dropped from schema and
+    // 2. if it has been re-added since then, this particular column was inserted before the last drop
+    private static boolean isDroppedColumn(Column c, CFMetaData meta)
+    {
+        if (meta.getDroppedColumns().isEmpty())
+            return false;
+        Long droppedAt = meta.getDroppedColumns().get(((CompositeType) meta.comparator).extractLastComponent(c.name()));
+        return droppedAt != null && c.timestamp() <= droppedAt;
+    }
+
+    private void removeDroppedColumns(ColumnFamily cf)
+    {
+        if (cf == null)
+            return;
+
+        Iterator<Column> iter = cf.iterator();
+        while (iter.hasNext())
+            if (isDroppedColumn(iter.next(), metadata))
+                iter.remove();
+    }
+
     /**
      * @param sstables
      * @return sstables whose key range overlaps with that of the given sstables, not including itself.
                     return null;
 
                 result = removeDeletedCF(cf, gcBefore);
-
             }
+
+            removeDroppedColumns(result);
         }
         finally
         {
                             data.addAll(cf, HeapAllocator.instance);
                     }
 
+                    removeDroppedColumns(data);
+
                     if (!filter.isSatisfiedBy(rawRow.key.key, data, null))
                         continue;
 
                     // cut the resultset back to what was requested, if necessary
                     data = filter.prune(data);
                 }
+                else
+                {
+                    removeDroppedColumns(data);
+                }
 
                 rows.add(new Row(rawRow.key, data));
                 matched++;

File src/java/org/apache/cassandra/db/marshal/CompositeType.java

         return null;
     }
 
+    // Extract CQL3 column name from the full column name.
+    public ByteBuffer extractLastComponent(ByteBuffer bb)
+    {
+        int idx = types.get(types.size() - 1) instanceof ColumnToCollectionType ? types.size() - 2 : types.size() - 1;
+        return extractComponent(bb, idx);
+    }
+
     @Override
     public int componentsCount()
     {