Package org.apache.sysds.runtime.instructions.spark

Class MatrixIndexingSPInstruction

java.lang.Object

org.apache.sysds.runtime.instructions.Instruction

org.apache.sysds.runtime.instructions.spark.SPInstruction

org.apache.sysds.runtime.instructions.spark.ComputationSPInstruction

org.apache.sysds.runtime.instructions.spark.UnarySPInstruction

org.apache.sysds.runtime.instructions.spark.IndexingSPInstruction

org.apache.sysds.runtime.instructions.spark.MatrixIndexingSPInstruction

All Implemented Interfaces:

LineageTraceable

public class MatrixIndexingSPInstruction
extends IndexingSPInstruction

This class implements the matrix indexing functionality inside CP.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.instructions.spark.SPInstruction

SPInstruction.SPType

Nested classes/interfaces inherited from class org.apache.sysds.runtime.instructions.Instruction

Instruction.IType

Field Summary

Fields inherited from class org.apache.sysds.runtime.instructions.spark.ComputationSPInstruction

input1, input2, input3, output

Fields inherited from class org.apache.sysds.runtime.instructions.Instruction

DATATYPE_PREFIX, FEDERATED_INST_PREFIX, GPU_INST_PREFIX, INSTRUCTION_DELIM, LITERAL_PREFIX, OPERAND_DELIM, SP_INST_PREFIX, VALUETYPE_PREFIX

Method Summary

Modifier and Type	Method	Description
org.apache.commons.lang3.tuple.Pair<String,LineageItem>	getLineageItem(ExecutionContext ec)	Obtain lineage trace of an instruction with a single output.
static MatrixBlock	inmemoryIndexing(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in1, DataCharacteristics mcIn, DataCharacteristics mcOut, IndexRange ixrange)
static boolean	isMultiBlockLookup(org.apache.spark.api.java.JavaPairRDD<?,?> in, DataCharacteristics mcIn, DataCharacteristics mcOut, IndexRange ixrange)	Indicates if the given index range and input matrix exhibit the following properties: (1) existing hash partitioner, (2) out-of-core input matrix (larger than aggregated memory), (3) aligned indexing range (which does not required aggregation), and (4) the output fits twice in memory (in order to collect the result).
static boolean	isSingleBlockLookup(DataCharacteristics mcIn, IndexRange ixrange)	Indicates if the given index range only covers a single blocks of the inputs matrix.
void	processInstruction(ExecutionContext ec)	This method should be used to execute the instruction.

Methods inherited from class org.apache.sysds.runtime.instructions.spark.IndexingSPInstruction

parseInstruction

Methods inherited from class org.apache.sysds.runtime.instructions.spark.ComputationSPInstruction

getOutputVariableName

Methods inherited from class org.apache.sysds.runtime.instructions.spark.SPInstruction

getGraphString, getSPInstructionType, getType, postprocessInstruction, preprocessInstruction, requiresLabelUpdate

Methods inherited from class org.apache.sysds.runtime.instructions.Instruction

getBeginColumn, getBeginLine, getEndColumn, getEndLine, getExtendedOpcode, getFilename, getInstID, getInstructionString, getLineNum, getOpcode, getOperator, getPrivacyConstraint, printMe, setInstID, setLocation, setLocation, setLocation, setLocation, setPrivacyConstraint, setPrivacyConstraint, toString, updateInstructionThreadID

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.sysds.runtime.lineage.LineageTraceable

getLineageItems, hasSingleLineage

Method Detail

processInstruction

public void processInstruction(ExecutionContext ec)

Description copied from class: Instruction

This method should be used to execute the instruction.

Specified by:

processInstruction in class SPInstruction

Parameters:

ec - execution context

inmemoryIndexing

public static MatrixBlock inmemoryIndexing(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in1,
                                           DataCharacteristics mcIn,
                                           DataCharacteristics mcOut,
                                           IndexRange ixrange)

isSingleBlockLookup

public static boolean isSingleBlockLookup(DataCharacteristics mcIn,
                                          IndexRange ixrange)

Indicates if the given index range only covers a single blocks of the inputs matrix. In this case, we perform a key lookup which is very efficient in case of existing partitioner, especially for out-of-core datasets.

Parameters:

mcIn - matrix characteristics

ixrange - index range

Returns:

true if index range covers a single block of the input matrix

isMultiBlockLookup

public static boolean isMultiBlockLookup(org.apache.spark.api.java.JavaPairRDD<?,?> in,
                                         DataCharacteristics mcIn,
                                         DataCharacteristics mcOut,
                                         IndexRange ixrange)

Indicates if the given index range and input matrix exhibit the following properties: (1) existing hash partitioner, (2) out-of-core input matrix (larger than aggregated memory), (3) aligned indexing range (which does not required aggregation), and (4) the output fits twice in memory (in order to collect the result).

Parameters:

in - input matrix

mcIn - input matrix characteristics

mcOut - output matrix characteristics

ixrange - index range

Returns:

true if index range requires a multi-block lookup

getLineageItem

public org.apache.commons.lang3.tuple.Pair<String,LineageItem> getLineageItem(ExecutionContext ec)

Description copied from interface: LineageTraceable

Obtain lineage trace of an instruction with a single output.

Specified by:

getLineageItem in interface LineageTraceable

Overrides:

getLineageItem in class ComputationSPInstruction

Parameters:

ec - execution context w/ live variables

Returns:

pair of (output variable name, output lineage item)