Prostate cancer is a heterogeneous disease whose progression is linked to genome instability. However, the impact of this instability on the non-coding genome and its three-dimensional organization to aid progression is unclear. Using primary benign and tumor tissue, we find a high concordance in higher order three-dimensional genome organization. This concordance argues for constraints to the topology of prostate tumor genomes. Nonetheless, we identified changes in focal chromatin interactions, typical of loops bridging non-coding cis-regulatory elements, and showed how structural variants can induce these changes to guide cis-regulatory element hijacking. Such events resulted in opposing differential expression of genes found at antipodes of rearrangements. Collectively, these results argue that changes to focal chromatin interactions, as opposed to higher order genome organization, allow for aberrant gene regulation and are repeatedly mediated by structural variants in primary prostate cancer.